Security incidents will happen. New types of exploits and attacks emerge frequently and those threats are becoming more numerous and diverse.  As security professionals, our job is to manage risk. (more about that here) We implement technical controls, establish sound policies, and take preventative measures in an attempt to make unacceptable losses as unlikely as possible and decrease the overall number of security incidents.  What we can not do is achieve a secure state.  There will be incidents and there will be loss.  Managing risk means planning what to do when an incident occurs so that we can minimize that loss.  This is where an effective incident handling plan comes in to play.  Performing incident response effectively is a complex undertaking and establishing a successful incident response capability requires significant planning and resources.

What DO you do when a security incident occurs?  PANIC!  The network intrusion detection sensors have alerted on suspicious traffic coming from your ecommerce website.  You start reviewing logs, and it appears that the application used for credit authorization has recently been altered and the machine is transferring files over FTP to somewhere outside of the country.  What do you do?  PANIC!

Take a deep breath.  You have thought about and planned for incident response.  There will be some loss, but the incident handling methodology you have laid out is about to minimize that loss.

There are traditionally 6 steps for incident handling.  These are:

1. Preparation
2. Identification
3. Containment
4. Eradication
5. Recovery
6. Review or Lessons Learned

Preparation and identification are your steady state.  These are what you’re doing all the time.  When an incident occurs, you snap in to action and save the day.  However, you must be prepared to do so.

I am going to focus on that first step preparation.  I think of this as less of a step in the process and more like the requisites you need to consider starting your incident handing process.  You need to have the people, tools, and plans in place.  Let’s get a little more specific about what you need on hand before you able to take care of an incident.

First, you need people.

Management – To be effective, you are going to need to invest a little money.  The tools cost money.  It will cost even more money to hire new people to handle incidents, or to take the time to train current staff.  When an incident occurs, the incident handling teams requires the authority to make certain high level decisions.  Management will need to provide budget, staff, and a pre-agreed upon formal robust structure to make important decisions.

Incident Handing Team (CIRT – Computer Incident Response Team) – Who will be doing the footwork?  You need a cross functional team with a lot of broad knowledge.  You also need technical people on the team with a deep understanding of your systems.  This should be a carefully selected and well trained group of people.  Break the team down in two parts.  Have a centralized technical core who will do the work, and a supporting cast of stakeholders who need to be aware of what is going on and will support the team with specific required knowledge and who can make the appropriate decisions.

Who should this team consist of?

• System and Network Administrators – Unless you are hiring or outsourcing your CIRT, this can your core technical team.
• Business managers – IT supports the business. If you think a system should be taken down, leave it up to the business.
• Human resources – Occasionally, an incident is the result of insider malicious activity. You may have to walk someone to the door.
• Representatives from the legal department – Occasionally, incidents end up in court.  The lawyers are going to want a hand in certain verbiage, and should be kept in the loop when it comes to notifying authorities.  When you take the person you just walked to the door to trial, that lawyer will want to have been involved from day one.
• Public relations – Occasionally, incidents are public.

Then, you need basic ground rules.

Incident reporting mechanisms – Make people feel comfortable reporting incidents, and have more than one way to report such as a website, phone number, and email address.

Privacy – If someone can argue in court that they had an expectation of privacy while using your system, some of your carefully collected evidence might not be admissible. Use log in warning banners. When people are using your systems, they need to be aware that they do not have that reasonable expectation of privacy.  If you have a log in message warning them that they may be monitored, that presumption of privacy has no grounds.

Note taking – While working an incident, you need to take meticulous notes.  This does two things.  People involved in incident handling tend to panic and rush to quickly get their systems back in a working state.  They tend to destroy or overlook vital evidence.  If you take the time to write detailed notes, you force yourself to slow down and think about what you are doing.  Also, if your case happens to go court, it will not be until months later.  If you have detailed notes, you are much less likely to forget important details.  These notes might even become evidence. They are more likely to be admissible if you have proof that the notes were not tampered with or altered.  Notepads with numbered pages, keeping your notes in a locked drawer, and a chain of custody around the written notes of an incident will help.

Contact information – Once you have a team, get laminated cards with the team’s phone numbers and encourage folks to keep it in their wallet.  When you need to call someone, you will save time not having to look up the number.

Communication Plans – When VoIP phones and email servers are compromised or down, you may have to use out of band communication.  Your team should be given cell phones.  If an extreme emergency, you might need to rely on ham radios.

War Room – You need a place to review evidence, talk about the incident, coordinate events, and handle communications.  Since you could be dealing with sensitive information, this should not be a war cube.  You need locked storage or filing cabinets.

Finally, you need some basic tools at the ready.

• Notebook – A handlers journal with numbered pages.  Other helpful resources like forms and templates for your notes.
• Camera – you should take digital still images of screenshots, the scene, etc…
• Evidence gathering accessories – chain of custody forms, evidence storage bags and tags
• Blank Media – CDs, DVDs, hard drives.  Hard drives need to have significant capacity.  They need to be 15% larger than the drive you’re copying, and you should have enough drives to make two copies of any disc you want to use as evidence.
• Computer forensic software – Helix
• Hub or Tap – If you need to sniff the traffic, using a tap and a laptop with Wireshark does the job quick.
• Computer forensic workstations or backup devices to create disk images, preserve log files, and save other relevant incident data.
• Packet sniffers and protocol analyzers to capture and analyze network traffic that may contain evidence of an incident
• Backup images of operating systems, applications, and data stored on secondary media
• Jump Bag – These tools should be ready to go at a moment’s notice.  Pre-pack a jump bag with everything you will need for an all night incident response session.  Pack a laptop loaded with appropriate software, cables, media, tools, software, flashlight, extra cell phone batteries, jumpers, pens, business cards, hardware write blockers, and even a change of cloths.

With these basics in hand, you can begin planning on how to handle the incident.  I’ll talk about the other states of incident response in upcoming entries.

Other Resources:
NIST 800-61: Incident Handling
SANS Incident Forms
CERT CIRT Handbook
Incident Response and Computer Forensics, Second Edition