24 — Incident Response

Level: Advanced · Time: ~22 min · Prerequisites: Lesson 23 — Detection Engineering and SIEM

Why this matters

Everything in this module so far has been about reducing the chance and the depth of a compromise. This lesson is about the hours when it has already happened and someone has to decide what to do next. Incident response is the discipline that converts a bad day into a contained one, and the reason it deserves a lesson of its own is that the decisions are made under pressure, with incomplete information, by people who have never practised them. An unexercised response plan is not a plan — it is a document that will be read for the first time while the file server is being encrypted. The material here is process, not tooling, which is why a small team with a written plan beats a large team improvising.

The mental model: event, incident, nuisance

Term	Meaning	What it demands
Event	something observable happened — a failed login, a blocked connection, a new service	recording it, and deciding whether it matters
Nuisance	an event with no security consequence — a user forgetting a password	closing it, and using it to tune the noise out
Incident	a confirmed or strongly suspected security compromise with business impact	the full lifecycle below, with people, decisions and a record

Most of the difficulty is telling the second row from the third. That judgement is what triage exists for, and the bias should be towards treating a confirmed detection as a possible foothold rather than a curiosity — a single alert on a workstation is rarely worth waking someone, and almost never worth ignoring until Monday.

The lifecycle

NIST SP 800-61, the standard incident handling guide, describes a lifecycle of preparation, detection and analysis, containment/eradication/recovery, and post-incident activity. Expanded for a small team, it looks like this:

  1. Preparation        before anything happens: plan, roles, access, tooling, baseline
        │
  2. Detection          triage, scope, timeline, preserve before you investigate
     and analysis
        │
  3. Containment        stop the bleeding now; keep the evidence; do not tip them off
        │
  4. Eradication        rebuild rather than clean; rotate every credential reached
     and recovery
        │
  5. Post-incident      lessons learned, dated actions, owned by a name

The phases are a checklist, not a waterfall. You will loop back: a scope discovery during recovery sends you back to containment, which is normal and expected.

Preparation: the artefacts you need before anything happens

Artefact	Why it exists	The failure without it
A written plan with named roles and named backups	nobody argues about who is in charge at 02:00	two people give the customer two different answers
An out-of-band contact list, kept offline	the incident may be that email and chat are compromised	you cannot reach anyone using the tools you are evacuating
A pre-agreed decision authority	who may shut a service down, who may approve payment, who speaks to customers	expensive decisions made by whoever happens to answer the phone
Credentials that do not depend on the systems under attack	you may need to log into something you have just isolated	the admin console is locked behind the compromised identity provider
Forensic tooling and blank media prepared in advance	collection must be possible immediately and legally sound	hours lost downloading tools onto a host in the middle of an incident
Legal, insurer and law-enforcement contacts	notification duties and coverage have deadlines	missed deadlines and an uncovered claim
A logging baseline	you must be able to answer "what changed"	you can see the attack and cannot see the intrusion
A recent, tested backup (Lesson 21)	recovery is usually the whole response	nothing to restore to, which turns an incident into a closure

The one-line summary: preparation is mostly a list of things that must not depend on the environment you are about to lose.

Detection, analysis and containment

Triage in the first hour

Scope before you fix. The question is not "what is this alert" but "how far has this gone". One alert on one workstation is a hypothesis; the same alert plus an unusual login on two servers is a different incident.
Establish a timeline. Pull events from the endpoint, the identity provider, the firewall and the resolver into one ordered view. This is where synchronised clocks (Lesson 22) pay off, and where an unsynchronised environment costs hours.
Distinguish the initial symptom from the initial access. A popped-up ransomware note is the symptom; the first access was probably weeks earlier, most likely a credential or an exposed service. Answering "when did it really start" changes who else is affected.
Preserve before you investigate destructively. Volatile data — memory, running processes, open network connections — disappears on shutdown and changes with every command you run. Collect it first or accept that you will never have it. Lesson 25 covers the mechanics; the discipline belongs here, and it is mostly the discipline of doing less.

[!WARNING] The instinct to "just reboot it and see if it comes back" costs evidence that cannot be recovered. Every restart overwrites memory, clears session state and may trigger the attacker's own cleanup. Isolate, collect, then act deliberately.

Containment: the real trade-offs

Choice	Isolating/blocking	Shutting down/disabling
The host	preserves memory, running processes and the chance to watch the attacker	destroys memory and volatile evidence, and can trigger tooling that dies with the process
The account	blocking sign-in keeps sessions and audit visible	disabling ends access but may tip off the attacker
The service	segmenting limits spread while keeping the business running	shutting it down guarantees safety and guarantees the outage

Practical decisions to make explicitly rather than by habit:

Short-term containment stops the bleeding: isolate the host, block the C2 destination, force a password reset on the account, add the malicious domain to the blocklist.
Long-term containment buys time for a clean rebuild: keep the attacker contained in a monitored segment while building a replacement environment.
Decide in advance who can authorise a business-level shutdown, because that call is the one that costs money and reputation, and it should not be made in a hurry by an engineer alone.
Do not tip off the attacker before you are ready. The moment you isolate a host or block a domain, the attacker knows. If you are not yet able to remove them everywhere, that early signal can trigger destruction — backup deletion, mass encryption — as a parting act.

Eradication, recovery and post-incident activity

Eradicate by rebuilding, not cleaning

For anything that was fully compromised, the professional default is a rebuild from a known-good source. Cleaning is attractive because it is faster, and it fails because you cannot enumerate what was changed: a handful of registry keys, a scheduled task, a modified service binary, a patched DLL, a new local administrator created under a name you did not look at.

Step	The detail that matters
Rebuild from known-good images	which means published, tested images and a way to apply current configuration, not a VM snapshot from before the change you needed last week
Rotate every credential the host could reach	including service accounts, API keys, database passwords, certificates and machine secrets — this is the step most often missed and the reason a second intrusion follows
Check for persistence before you reattach	scheduled tasks, services, autoruns, new accounts, delegated permissions, mailbox rules, OAuth application consents
Recover in a monitored way	heightened logging, a shorter alert threshold, the endpoint agent verified as healthy, and an explicit watch for the same indicators
Watch for the second intrusion	a large share of organisations are re-compromised within weeks because one credential or one persistence point survived

Post-incident activity

Run a lessons-learned session within a couple of weeks, while memories are fresh, with the whole team and a written output. The five questions that produce useful answers:

How did they get in, and can we say that with evidence rather than assumption?
What was the first thing we could have detected, and why did we not?
What slowed us down? Missing contacts, unknown asset owners, no logs, unclear authority, someone on holiday.
What surprised us?
Which single control would have prevented or shortened this — and is it now on a dated list with a name against it?

Findings become actions with an owner and a date, or they become opinions. Track them in the same way you track any other work, and review whether they happened.

Communication, notification and the record

Incident response is partly a communications exercise, and this is the part nobody documents until the day it is needed.

Audience	What they need	The mistake
Internal leadership	impact, current containment, decisions required, next update time	technical detail nobody can act on, and no clear ask
Staff	whether to stop working, what to avoid, who to contact	silence, which produces rumour
Customers and partners	what happened, what it means for them, what to do	either saying nothing or over-sharing before facts are established
Regulators	notification within a legal deadline, factual and specific	missing the deadline because nobody had the contact
Insurer	immediate notification per the policy, plus the required evidence	discovering afterwards that the policy required notification within 24 hours
Law enforcement	a decision, not an automatic step	treating it as either mandatory or pointless without weighing the trade-offs

On the deadlines, be precise but do not take this lesson as legal advice. Under the EU General Data Protection Regulation, a personal data breach that poses a risk to individuals must be notified to the supervisory authority — the CNIL in France — without undue delay and within 72 hours of becoming aware of it, with notification to affected individuals where the risk is high. Other European rules and sector regulations add their own, sometimes shorter, clocks for organisations in scope. Get legal advice early rather than interpreting this paragraph: the practical failure is not getting the analysis wrong, it is discovering the deadline a week after it passed.

Who leads, and how decisions are recorded. Name one incident lead who owns the process and the decisions, with a technical lead doing the hands-on work. Keep a simple incident log — timestamp, decision, who made it, why — in one place, updated as you go. It is the raw material for the regulator, the insurer, the customer explanation and the lessons-learned session, and it is impossible to reconstruct afterwards.

Evidence handling in one paragraph. Gather volatile data before non-volatile, keep an unmodified copy, record who touched what and when, hash the images you take, and work from copies rather than originals. Get the order of volatility and the chain of custody right from the first hour, because evidence collected carelessly is not merely weak — in some contexts it is unusable. Lesson 25 covers the mechanics.

Ransomware, and the first hour, day and week

Ransomware principle	The reasoning
Do not pay without legal and law-enforcement advice	payment funds criminal operations, does not guarantee decryption, may breach sanctions rules, and is often followed by a second demand
Assume data was taken even when only encryption is visible	exfiltration before encryption is standard, so the "no data was stolen" position is unsafe without evidence
Restore from the known-good backup, not from the attacker's tooling	the decryptor is unreliable and running it means running the attacker's code on your estate
Identify and remove the access route before restoring	restoring while the entry path is open is how organisations pay twice
Sequence it: contain, verify backups, rebuild the entry path, rotate credentials, restore in a monitored way	recovery in the wrong order is how the second encryption happens

The one-page checklist

First hour

Confirm and record the time of detection; start the incident log.
Notify the incident lead and the decision authority; open the out-of-band channel.
Isolate affected hosts rather than shutting them down.
Preserve volatile evidence before anyone starts "having a look".
Change nothing that tips off the attacker while you establish scope.

First day

Scope the incident: which accounts, which systems, which data.
Contain: block destinations, disable or block accounts, segment the affected network.
Verify backups exist, are offline or immutable, and that their credentials are not the compromised ones.
Notify per plan: leadership, insurer, legal, and any partner with an obligation.
Start the timeline from collected evidence, not from memory.

First week

Rebuild rather than clean, from known-good images.
Rotate every credential reachable from the compromised hosts, including service accounts and API keys.
Restore in a monitored order, with heightened logging and a second-intrusion watch.
Handle notification deadlines with legal advice — the 72-hour clock starts when you became aware.
Hold the lessons-learned session; write dated actions with named owners.

Attack it / Defend it

The attack	How it works	The control that stops it
Ransomware	encrypts production data and destroys backups	offline or immutable backups, allowlisting, fast isolation
Double extortion	exfiltrates data, then encrypts, to force payment	egress monitoring, least privilege, a plan that assumes theft
Attacker tips the second you act	observes isolation and destroys evidence or data	sequenced containment: contain widely before acting visibly
Re-compromise after recovery	a rotated-nowhere credential or surviving persistence	rotate everything reachable, verify persistence removal, monitor closely
Living-off-the-land	native tools so no malware is found to remove	behavioural detection, and rebuilding rather than cleaning
Evidence destruction	clears logs, wipes memory by rebooting	central log collection, isolation instead of power-off, immediate preservation
Communication pressure	threats and deadlines aimed at your negotiators	a pre-agreed decision authority and legal guidance, not improvisation
Compliance clock abuse	the breach is discovered late, shortening your response window	detection coverage from Lesson 23, so awareness starts when the incident does
Insurer friction	policy conditions unmet at claim time	knowing the notification terms before the incident, not after

Key takeaways

An unexercised plan is not a plan. Rehearse the contacts, the isolated-host procedure and the restore before you need them.
Preparation is everything that does not depend on the systems you are about to lose — offline contacts, independent credentials, prepared tooling, a logging baseline.
Isolate, do not power off. Memory and volatile evidence vanish on shutdown, and so does your best chance of understanding the intrusion.
Rebuild rather than clean, and rotate every credential the compromised host could reach. Surviving persistence is why organisations are attacked twice.
Record decisions as you make them. The incident log serves the regulator, the insurer, the customer and the lessons-learned session, and it cannot be reconstructed later.
Deadlines are real. Under GDPR, a personal data breach generally triggers notification within 72 hours of awareness — get legal advice rather than guessing.

Check yourself

Why is isolating a compromised host better than shutting it down, and what specifically do you lose by powering it off?
You have contained a compromised server and want to remove the attacker's access. What must be rotated before you consider the host clean, and what will happen if you only reset the user's password?
What decision authority should be agreed in advance, and why is the alternative so expensive during an incident?
Why must you assume data was stolen even when the only visible evidence is encryption?
A personal data breach is confirmed at 11:00 on a Friday. What clock has started, who needs to be told, and what should you not do yourself?