23 — Detection Engineering and SIEM
Level: Advanced · Time: ~22 min · Prerequisites: Lesson 22 — Logging and Monitoring Foundations
Why this matters
Logging gave you the raw material; this lesson is about turning it into decisions. A SIEM is the industry's name for the machinery that does it, and it is also the most reliably disappointing purchase in security, because organisations buy the machinery and forget that the detection comes from rules and people. This is the lesson where you learn to write a rule that finds a real intrusion without drowning your team, how to tell a good rule from a noisy one, and why the project usually dies at the point where somebody decides not to read the alerts any more.
Click the diagram to open it full size.
What a SIEM is, and what it is not
| A SIEM does | A SIEM does not |
|---|---|
| collect from many sources into one place | generate detections by itself — the rules and the analysts do that |
| normalise different formats into common fields | understand your business without enrichment you feed it |
| enrich with asset, user and threat data | replace endpoint telemetry, which it only holds a copy of |
| correlate events across sources into one finding | decide whether an alert is real; that is triage |
| store for search and retention | replace backups, or the forensic detail on the original host |
| alert a human, and provide search for investigations | fix a collection gap — no data in, no detection out |
The last row is the whole lesson of the previous one. A SIEM fed by three log sources produces detections covering three log sources.
The open-source equivalent
You do not need a commercial platform to build this, and Lesson 35 walks through the components in detail. The pragmatic shape for a small team:
| Layer | Open-source options |
|---|---|
| Host and identity collection and detection | Wazuh — an agent plus a manager, the usual starting point |
| Network visibility | Suricata for signature-based network detection, Zeek for connection and protocol metadata |
| Storage and search | OpenSearch or Elasticsearch |
| Dashboards | the Wazuh or OpenSearch dashboards, or Grafana |
| Rule portability | Sigma, a vendor-neutral rule format for Windows and Linux telemetry, converted to the target platform with pySigma |
| Endpoint interrogation and collection | osquery for host state queries, Velociraptor for targeted collection across a fleet |
| Case handling | TheHive with Cortex for case management and enrichment |
The trade-off is honest: an open-source stack costs licence money you do not pay and engineer time you do. For a small team the deciding question is not "which is better" but "which will one person actually maintain".
The anatomy of a detection rule
Every rule, whatever the platform, answers the same questions.
| Component | The question it answers | Why beginners get it wrong |
|---|---|---|
| What it matches on | a single event, a sequence, a threshold, or a pattern across sources | they write signatures for things that are better caught by behaviour |
| The fields it needs | which log fields must exist and be populated | they write the rule before checking the field is collected — Lesson 22 comes first |
| Conditions | the logic that combines the selections | too broad, so it fires on everything; or too narrow, so it never fires |
| Severity | how urgent this is if true | everything is marked critical, so nothing is |
| ATT&CK mapping | which tactic and technique this covers | omitted, so coverage cannot be measured |
| Response guidance | what the analyst should do first | omitted, so the first five minutes are spent guessing |
A portable rule in the Sigma style, with the sections labelled:
title: Office application spawning a command interpreter
status: experimental
description: A document reader or spreadsheet creating a shell or scripting host.
logsource:
product: windows
category: process_creation
detection:
selection_parent:
ParentImage|endswith:
- '\WINWORD.EXE'
- '\EXCEL.EXE'
selection_child:
Image|endswith:
- '\cmd.exe'
- '\powershell.exe'
- '\wscript.exe'
- '\mshta.exe'
condition: selection_parent and selection_child
falsepositives:
- Document automation built by the finance team
level: high
tags:
- attack.execution
- attack.t1059.001
Read it as a sentence: when a Microsoft Office process starts a command interpreter, alert, at high severity, and map it to the execution tactic. The falsepositives section is not decoration — it is the tuning instruction for the person who reads the alert next month.
The four rule archetypes
| Archetype | What it matches on | Concrete example | Trade-off |
|---|---|---|---|
| Signature | a known-bad artefact | a file hash, a known malicious domain, a specific user-agent string | precise and cheap, but weakest — the attacker changes one byte or one character and it is gone |
| Behavioural | an action that is suspicious regardless of which tool performed it | a workstation creating a scheduled task that runs a script from a user temp directory | survives tool changes, needs tuning, produces real false positives |
| Correlation | two or more events that are individually benign | a successful login from a new country immediately followed by a mass file read | high fidelity when the combination is genuinely unusual; useless if the individual events are not collected |
| Threshold / anomaly | a rate or pattern change | twelve failed logins across eight accounts in two minutes | catches spraying that never trips lockout; needs a baseline and is noisy during change |
The four are complementary, not ranked alternatives. Use signatures where you have confirmed bad artefacts from your own incidents, behaviour where you want to survive tool changes, correlation for the high-value combinations, and thresholds for the low-and-slow attacks designed to stay under other rules.
Write against behaviour, not indicators
Lesson 7 introduced the pyramid of pain, from hash values and IP addresses at the shallow end up to tactics, techniques and procedures at the painful end. Detection engineering is that pyramid applied: a rule that matches a hash is defeated by recompilation; a rule that matches an IP address is defeated by a new server; a rule that matches a domain is defeated by a new registration; but a rule that matches the action — a credential dump, a scheduled task from a temp directory, a login from a new country — forces the attacker to change how they work. Write down the pyramid position of every rule you keep, and retire the ones that only live at the shallow end.
The operational reality: where SIEM projects die
Alert fatigue is the primary failure mode
A rule that fires four hundred times a day is worse than no rule at all, because the analyst stops reading. The volume is invisible on the day the rule is written; it becomes the reason the project is abandoned three months later. Measure every rule by its alerts per day, and treat any rule above a handful as broken until proven otherwise.
Tuning is continuous, and there are only five actions. Learn them:
| Tuning action | When to use it |
|---|---|
| Add context | the alert is correct but uninformative — enrich with asset owner, user role, last login |
| Raise the threshold | it fires often on legitimate volume — from 1 event to 5 in a window |
| Suppress the known-benign | a specific, identifiable, legitimate case — with a documented reason and an expiry date |
| Split the rule | one rule covers two very different risks with different severities |
| Retire it | nobody has ever acted on it — deletion is a valid engineering decision |
False positives versus false negatives are a dial, not a mistake. Which way to lean depends on the asset: on a domain controller, tolerate noise and catch everything; on a fleet of laptops, a rule nobody reads protects nothing. State the lean explicitly so the next engineer understands the intent.
Triage: what an analyst does in the first five minutes
- Validate the alert is real — is the underlying event actually present and correctly parsed, or is it a broken rule?
- Identify the asset and the user — what is this machine, who is this person, and does the alert make sense for both?
- Check whether the action was expected — a change ticket, a scheduled deployment window, a known administrative tool, an approved script.
- Look for related events — what happened on that host and that account before and after, across sources. This is what correlation is for.
- Decide and record — true positive, false positive, or needs escalation, written down with the reasoning. A decision not recorded is a decision that will be made again next week.
Then severity and paging: informational for records nobody reads, low for the weekly review, medium for the working day, high for immediate attention, critical for whoever is on call at three in the morning. The number of things that should page someone is small. Be ruthless about it, because a pager that cries wolf is switched off.
Detection as code
Rules are software and should be treated as such:
| Practice | Why |
|---|---|
| Rules in version control | you can see who changed what, when, and revert a bad change |
| A review before a rule is enabled | four eyes catches the field that does not exist |
| A test case per rule | the rule is verified to fire, and to not fire on a known-benign sample |
| A documented reason for every suppression | otherwise suppression becomes a permanent blind spot nobody remembers |
| An owner and a review date per rule | rules rot as the environment changes |
Measuring: coverage and four numbers
Map your rules against the fourteen ATT&CK tactics — reconnaissance, resource development, initial access, execution, persistence, privilege escalation, defence evasion, credential access, discovery, lateral movement, collection, command and control, exfiltration, impact — and colour each one by whether you could detect anything at all in that phase. The result is usually confronting: strong on execution and persistence, empty on discovery and lateral movement. That gap is your roadmap.
Then track four numbers, and only four:
| Number | What it tells you |
|---|---|
| Alerts per day | whether volume is manageable, per rule and in total |
| Percentage triaged | how much of the output is landing on a human |
| Time to triage | how long a real alert waits before someone looks |
| Rules retired | whether the content is being maintained or only accumulated |
Start with five alerts
Do not build a hundred rules. Build these five properly, tune them for a month, and only then expand. Each one is high-signal, each has a benign explanation you will meet, and each is worth investigating anyway.
| Alert | Why it is high-signal | The usual benign explanation |
|---|---|---|
| A new privileged account or group membership | privilege change is an attacker's objective, and legitimate changes are rare and ticketed | a genuine new starter, an approved project, a change window |
| MFA disabled or a factor re-enrolled for a user | frequently the first step in account takeover, and almost never legitimate without a ticket | a helpdesk reset for a lost phone — verify against the ticket, and look for a login from a new location just before |
| Successful authentication from a new country, or impossible travel | a stolen credential being used from where the attacker is | genuine travel, a VPN egress in another region, a cloud service authenticating on the user's behalf |
| A new service, scheduled task or autorun entry created | persistence arriving on a host | software installation, an updater, a management agent, a legitimate script deployment |
| An outbound connection burst to a new domain | command and control or exfiltration in progress | a new SaaS tool, an advertising or telemetry domain, a misconfigured application |
[!TIP] For each of these, write the response guidance into the rule itself: what to check, who to contact, and what "this is fine" looks like. The analyst at three in the morning is not the person who wrote the rule.
Test the rule before you trust it
A rule that has never fired is a hypothesis. Verify it in the same way you verified logging:
- Generate the benign-but-similar event on purpose in your own lab — create a scheduled task, sign in from a phone on mobile data, add a test account to a test group.
- Confirm the alert appears with the right severity, the right fields and the right guidance.
- Confirm the suppression you added for the known-benign case does not also hide the real thing — deliberately create an event that should still fire and check that it does.
- Note the date you tested it. Rules decay as environments change, so this becomes an annual exercise.
Attack it / Defend it
| The attack | How it works | The control that stops it |
|---|---|---|
| Alert flooding | generates noise so the real alert is missed | per-rule volume limits, tuning, prioritising high-signal over high-volume |
| Living-off-the-land | uses built-in tools, defeating signatures | behavioural rules on command lines and parent processes |
| Slow and low brute force | stays under lockout and threshold rules | threshold rules across accounts, not per account |
| Rule-based evasion | changes an indicator after learning your rule | detections written against behaviour, plus periodic review of what fired |
| Telemetry blinding | disables the agent or clears logs | health monitoring, central collection, alert on silence |
| Indicator reuse after burn | retires a domain or tool your rule matches | correlation rules on combinations rather than artefacts |
| Alert-driven social engineering | floods you with a fake incident to distract | clear severity and paging criteria, and one person deciding priorities |
| Phishing-resistant threshold gaming | uses valid credentials so no failure events are generated | success-based detections: new location, new device, new privilege |
| Suppression abuse | an attacker who knows your suppression rules hides inside them | every suppression documented, dated, scoped and reviewed |
Key takeaways
- A SIEM does not detect anything. It collects, stores and searches; the rules and the analysts produce the detection.
- A rule's job is to be read. A rule that fires four hundred times a day has negative value because it trains the team to ignore the console.
- Signatures are for confirmed bad artefacts; behaviour is for everything else. Behaviour costs more to write and survives the attacker changing tools.
- Tuning has five moves — add context, raise the threshold, suppress known-benign, split the rule, retire it — and retiring is a legitimate engineering decision.
- Measure coverage by ATT&CK tactic and track four numbers. Anything else is reporting rather than engineering.
- Five well-tuned alerts beat a hundred untested rules. Test each one by making a benign version of the event happen.
Check yourself
- Your SIEM is feeding on endpoint logs alone. Which ATT&CK tactics can you genuinely detect, and which are out of reach without network and identity logs?
- A rule fires 300 times a day and nobody reads it. Name the five tuning actions and which one you would apply first, and why.
- Why is a behavioural rule for "office application spawning a shell" more durable than a rule matching a known malicious file hash?
- An analyst gets an alert for a new privileged account at 02:00. What are the first five minutes of triage?
- You want to test a new rule. Describe how you would verify both that it fires and that your suppression does not hide the real event.