26 — Threat Hunting
Level: Advanced · Time: ~22 min · Prerequisites: Lesson 25 — Digital Forensics Basics
Why this matters
Everything you have built so far reacts. Rules fire, alerts pop, someone triages. Hunting is the deliberate opposite: you start with a question, go looking for evidence in data you already collect, and either find an intruder your detections missed or prove, with a record, that a specific technique is not present here. It exists because detections are written for behaviour somebody already knew about, because attackers change their tooling precisely to stay under those rules, and because dwell time — the gap between compromise and discovery — is measured in weeks in most organisations. If an intruder is already inside and no rule has fired, only a hunt will find them.
This lesson is also the honest one. Hunting is the first activity a small team drops when it is busy, so the design goal is not "become a hunter". It is to run hunts that leave something permanent behind: a detection, a baseline, or a documented negative you never have to redo.
The mental model: hunting is a loop, not a dashboard
Hunting is proactive, hypothesis-driven investigation to find what your detections missed. It is not alert triage — that is reactive and belongs to Lesson 24. It is not vulnerability scanning, which asks what could be exploited rather than what is happening. And it is not reading dashboards until something feels odd, which is how an afternoon disappears without a single question being answered.
| Step | The question you are answering | The artefact you produce |
|---|---|---|
| 1. Form the hypothesis | what specific attacker behaviour could be present here, and why do I believe it | one written sentence |
| 2. Identify the data | which log or telemetry source would confirm or refute it | the source, its retention, and whether it really covers the period |
| 3. Query it | what would this look like, with the false positives included | a query, and the output |
| 4. Compare to baseline | is this different from normal here, not normal in general | a judgment: anomaly or expected |
| 5. Conclude | found something → incident (Lesson 24); found nothing → a validated negative | a decision, not a feeling |
| 6. Convert it | how do I never hunt this exact thing again | a new detection, or the query saved into the regular schedule |
Step 6 is the one that pays. A hunt that finds nothing and leaves no detection has consumed a day and returned nothing; the same hunt that leaves a rule behind has permanently improved your coverage. Treat "what detection does this leave behind?" as part of step 1, not an afterthought.
The hypothesis is the unit of work. Compare:
| Bad hypothesis | Why it fails | Better version |
|---|---|---|
| "Let us query everything and see what looks weird" | no answer is defined, no data source is named, no conclusion is possible | — |
| "We think an attacker could be using remote management tools to run commands on servers" | names a behaviour, a target group and a reason; the data is known | confirmable, refutable, and it becomes a detection |
Where good hypotheses come from: the ATT&CK tactics you have no coverage for (Lesson 7 gives the structure); a threat intelligence report about your sector that names a technique; a suspicious alert that was closed because nobody had time to resolve it; an anomaly somebody noticed in passing ("why is the finance server talking to a hosting provider?"); and the estate's own history — techniques that were seen in a previous incident.
Hunting indicators versus hunting behaviours. Indicators — an IP address, a file hash, a domain — are fast to search and brief to matter: the attacker changes them in an afternoon. Behaviours — a process spawning a shell, a host contacting an unfamiliar destination on a schedule, a new service that survives reboots — are slow to find and durable. The pyramid of pain from Lesson 7 is the economic argument: the higher up the pyramid you hunt, the more it costs the attacker to adapt.
| Indicators | Behaviours | |
|---|---|---|
| Time to write the hunt | minutes | hours to days |
| Useful life of the result | days | months to years |
| Skill required | low | the real craft |
| What it leaves behind | a blocklist entry | a detection that fires on intent, not on one attacker's infrastructure |
Three worked hunts
Each hunt below follows the same shape: hypothesis, data source, query logic in words, false positives, and the detection to leave behind. The specific queries depend on your platform, so the logic is written out rather than given as a product-specific query.
Hunt 1 — Anomalous authentication
Hypothesis. An attacker holding valid credentials logs in from a source, at a time, or to a system that this account has never used before.
Data sources. Successful authentication events — Windows security event 4624 with its logon type, Accepted lines in /var/log/auth.log or the journal, VPN and RADIUS records, and cloud identity sign-in logs. Failed attempts (4625, Failed password) are the supporting evidence, not the hunt itself.
Query logic. For each account, build a 90-day baseline of source address, hour of day, destination host and logon type. Then examine today's successful logons and count how many of those four attributes are new for that account. One novelty is common and boring — a laptop on a hotel network. Three novelties on the same logon is a finding.
False positives. Mobile and home connections change address daily, so treat a new address as weak and a new country as strong; new starters and role changes; a service account moved to a new host with a change record; a person working late through a deadline; a daylight-saving shift making one week's activity pattern look different.
Detection to leave behind. An interactive or remote-desktop logon (logon types 10 or 3 with an administrative account) to a server that account has never logged on to before. Separately, any successful logon from a country where you have no staff and no infrastructure.
Hunt 2 — Periodic outbound connections (beaconing)
Hypothesis. Malware on an internal host is calling home on a regular schedule and receiving commands.
Data sources. Proxy logs, firewall egress logs carrying duration and byte counts, rich network metadata (Zeek conn.log), DNS query logs with the client address, and flow records.
Query logic. Group outbound connections over seven days by (internal host, destination). For each pair, measure the intervals between consecutive connections: a low standard deviation with a mean between roughly 30 seconds and an hour is machine-like, and humans are not machine-like. Then look for the three accompanying shapes — many connections carrying very small and very consistent payloads (a check-in), a few very long-lived sessions to one destination (a tunnel), and one internal host resolving the same domain thousands of times a day with long, random-looking subdomain labels (DNS as the channel).
For each (internal host, destination) pair over seven days:
n = number of connections
gaps = the time between consecutive connections
jitter = stdev(gaps) / mean(gaps)
if n > 50
and jitter < 0.2 # machine-like regularity, not human
and 30s < mean(gaps) < 3600s
and mean(bytes_sent) < 2000 # a check-in, not a download
then candidate_beacon(host, destination)
False positives. Software updaters, browser telemetry, operating system update checks, chat and collaboration clients, backup and monitoring agents, and NTP all produce perfectly regular outbound traffic. They are beacon-shaped by design. This is why the signal is not periodicity alone but periodicity to a destination you cannot justify — which requires an egress policy to compare against.
Detection to leave behind. An alert on any internal host whose destination is outside the approved egress list and whose connection count exceeds a threshold in a day; plus a scheduled weekly query that ranks host–destination pairs by interval regularity and is reviewed rather than alerted on.
Hunt 3 — Persistence abuse
Hypothesis. Somebody has established a way to survive a reboot: a scheduled task, a service, an autorun, a cron entry, a systemd unit or a remote management agent that nobody in IT installed.
Data sources. The persistence locations themselves, with their creation and modification times — Windows event 4698 (scheduled task created) and 7045 (service installed), Sysmon file-create and registry-value-set events, systemd unit files, cron directories, autoruns, and file integrity monitoring such as AIDE or Wazuh over those paths.
Query logic. Enumerate every persistence location the operating system offers, capture each entry with its timestamp and what it points to, and diff that list against the same list from a month ago. For anything newer than the last change window that has no change record behind it, look at the target: a path under /tmp or a user profile, an unsigned binary, a random-looking directory name, a command line that downloads something, or an interpreter (powershell.exe, cmd.exe, /bin/bash, python) used as the target of a service.
False positives. Legitimate installers create services and tasks; monitoring and backup agents do the same; operating system patching rewrites unit files constantly; and managed remote-support agents pushed by IT are indistinguishable in shape from a remote access tool an attacker installed. That last point is the reason to keep a written inventory of the remote management agents you have approved, with their destinations.
Detection to leave behind. File integrity monitoring on the persistence directories — /etc/systemd/system, /etc/cron.d, /var/spool/cron, C:\Windows\System32\Tasks and the Run keys — plus an alert on task or service creation by any process that is not your deployment tool.
Two controls make this whole hunt cheaper: application control (allowlisting what may execute) means a new binary cannot run at all, whatever entry points to it, and file integrity monitoring catches the change whether or not it runs.
Making hunting operational
| Question | Practical answer |
|---|---|
| How often? | One two-hour hunt a week is a realistic small-team cadence; one full day a month for a deeper one. Consistency beats intensity |
| Who does it? | Whoever investigates alerts, paired with the person who knows the estate — a hunt with no knowledge of what normal looks like produces only false positives |
| How do you record a hunt? | One page: hypothesis, data source and date range, the query, time spent, conclusion, and the detection or baseline created. File it even when it finds nothing |
| What happens when you find something? | It stops being a hunt and becomes an incident (Lesson 24). Preserve first (Lesson 25) |
| How do you know it is working? | Count of hunts converted to detections, and detection coverage of ATT&CK techniques — not hunts run |
The maturity ladder, so you know where you are rather than where marketing says you should be:
| Level | What it looks like | What is missing |
|---|---|---|
| 0 — ad-hoc | someone pokes around after an interesting alert | repeatability, records, anything permanent |
| 1 — scheduled hunts from a list | a written list of hypotheses, one attempted per week, results recorded | data maturity: the answer is often "we do not have that log" |
| 2 — data-driven | you can answer "has this host ever…" — baselines exist and the data is retained long enough to look back | hypotheses still come from a list rather than from your own gaps |
| 3 — intelligence-driven | hypotheses come from ATT&CK coverage gaps and reporting relevant to your sector, every hunt ends in a detection, and the detection book is reviewed | this is a genuine programme, and few small organisations reach it |
Five hunts to start with, all answerable with data a defended network usually already has:
| Hunt | Hypothesis | Detection it leaves behind |
|---|---|---|
| 1 | An administrative account logged on to a server it has never used | first-use-of-account-on-host alert |
| 2 | An internal host connects out to a destination outside the egress list | egress exception alert |
| 3 | A new scheduled task, service or cron entry appeared outside a change window | file integrity monitoring on persistence paths |
| 4 | An account created outside the joiner process | alert on account creation events (event 4720, useradd) |
| 5 | A host resolved an unusual volume of names, or names never seen before in the estate | new-domain and query-volume alert from the resolver logs |
[!IMPORTANT] Hunting is the first thing cut when a small team is busy, and it should be. That is precisely why converting each hunt into a detection matters more than the hunt itself: the detection keeps working in the weeks when nobody hunts, and it works at 03:00 when nobody is awake. Judge your hunting programme by the detections it leaves behind, not by the hours it consumed.
Attack it / Defend it
| The attack | How it works | The control that stops it |
|---|---|---|
| Living off the land | only built-in system tools are used, so no signature matches | process lineage baselines; hunting parent–child pairs and unusual command lines |
| Valid accounts | stolen credentials produce no malware at all | authentication baselining, impossible-travel and first-use detections, MFA |
| Beaconing over allowed protocols | command and control hides inside HTTPS or DNS to a destination that looks ordinary | egress allowlists, DNS query logging with client attribution, interval-regularity queries |
| Persistence via legitimate mechanisms | a task, service or remote management agent nobody noticed | file integrity monitoring on persistence paths, application control, an approved-agent inventory |
| Slow and low activity | so little traffic that no volume threshold triggers | long-baseline comparisons and rare-event queries instead of thresholds |
| Timestomping and log tampering | local records are altered so a hunt finds nothing on that host | ship logs off-host in real time, alert on log clears, correlate with network-side data |
| Reusing one technique across the estate | the same tooling on many hosts, which is exactly what makes it findable | convert the first finding into a detection and query the whole estate for it immediately |
| Hiding in alert noise | the operator drowns the team in low-value alerts so real behaviour is not examined | tune detections down, and hunt the alert-free behaviour that should have fired |
Key takeaways
- A hypothesis is the unit of work. "We think X could be happening, so I will look in Y for Z" is a hunt; "let us query everything" is not.
- Found nothing is a result, but only if it is written down — a validated negative stops the same question being asked twice.
- Behaviours outlast indicators. Hunt the technique, not the address.
- Every hunt ends with a detection or a documented negative. The detection is what keeps working when nobody hunts.
- Data beats skill. Most hunts fail at "we do not retain that log", which is fixed by Lesson 22's work, not by better analysis.
Check yourself
- Rewrite this into a usable hypothesis: "let us look at network traffic for anything suspicious."
- You hunt for beacons and find that one host makes 1,440 connections a day to the same destination at exactly 60-second intervals. Name three benign explanations and the one question that distinguishes the malicious case.
- Why is the detection left behind more valuable than the hunt itself in a small team?
- Which artefacts would tell you whether a scheduled task was created last night, and what would you compare it against?
- Your hunt finds nothing, but you discover that the relevant log is retained for only 24 hours. What is the correct conclusion to record?