38 — Build Your Blue-Team Lab

Level: Intermediate · Time: ~25 min · Prerequisites: Lesson 37 — Build Your Home Security Lab

Why this matters

A lab that only collects logs is a filing cabinet. The blue team's product is a loop: telemetry is collected, a detection fires, an alert is triaged, and the investigation produces a change — in a rule, in a configuration, or in a documented acceptance of risk. If any one of those four stages is missing, you have installed software rather than built a capability, and Lesson 23 has already told you how that ends. This lesson assembles the open-source components from Lessons 31 to 35 into that loop on the range you built in Lesson 37, and then puts it under test so you know the loop closes.

The mental model: the loop, not the log store

Stage	The question it answers	The artefact it produces
Collect	is the event even reaching me, from every asset that matters?	a normalised record with an asset, a user and a timestamp
Detect	does a rule turn that record into a finding?	an alert with a severity, a reason and an ATT&CK mapping
Triage	is this real, and how urgent is it?	a decision, plus the evidence that justified it
Change	what now prevents, detects or accepts this?	a tuned rule, a configuration change, or a logged acceptance with an owner

The circle is the point: the change feeds back into collection and detection quality. A team that collects but never changes anything is paying for storage; a team that writes rules but never tests them is publishing rumours.

The components, and what each contributes

Component	Example (open source)	What it contributes	Where it was covered
Host agent	Wazuh agent	process, authentication, file and configuration events from each VM	Lessons 32 and 35
Manager	Wazuh manager with its rules and decoders	storage, decoding, correlation and alerting in one place	Lesson 23
Network sensor	Suricata, in IDS mode first	rule-based detection on the wire, visible at the lab gateway	Lesson 31
Rich network logging	Zeek, or Suricata's own event records	structured connection, DNS and HTTP records you can query	Lesson 31
Operating system audit	`auditd` on the Linux targets; Sysmon plus audit policy on the Windows target	process execution with arguments, file writes, privilege use	Lesson 22
DNS resolver that records	Unbound with query logging, or a Pi-hole-style resolver	a per-host record of every name looked up	Lesson 34
Honeypot	OpenCanary or Cowrie	an alert on any interaction, because nothing legitimate touches it	Lesson 34
Dashboard	the Wazuh dashboard, OpenSearch dashboards or Grafana	the interface where a human actually looks at any of the above	Lesson 35

Reference architecture for the lab:

        [ home network ]  ── one management URL, nothing else ──▶ blue-team VM
                                                                 (manager + dashboard
                                                                  + resolver + honeypot)
   lab gateway ──────────── lab firewall VM
        │  (span/mirror or a second interface)
        └─ Suricata (IDS) + Zeek  ── logs ──▶  manager

   attack VM ──┬── Windows target   (agent, Sysmon, audit policy)
               ├── Linux target     (agent, auditd, vulnerable web app)
               └── honeypot         (no agent; any traffic is the alert)

Everything in that diagram fits on one machine when the event volume is a handful of VMs and a few scans per session. Small volume is the lab's great advantage: you can run the full pipeline and still read every alert yourself.

The build sequence, in the order that avoids rework

Install and patch the blue-team VM on the lab network, with enough memory to hold the manager and the dashboard comfortably.
Deploy the manager and confirm the web interface answers on your management connection — and only there.
Enrol one agent on one target, then confirm an event from that host appears in the interface. One agent, proven, before you touch the second host.
Add the network sensor on the interface that sees lab traffic, in IDS mode, and confirm it writes events.
Confirm name resolution and a web request are recorded. Generate a DNS lookup and an HTTP request from a target, then find both in the records.
Add the audit rules on the Linux target and the equivalent endpoint logging on Windows, then confirm a file write is logged.
Only now add the extras — resolver filtering, dashboard, honeypot — one at a time, each with its own confirmation.

The proof that the pipeline works is a deliberate failure, not a green dashboard:

# On a lab target: fail an authentication on purpose
ssh labuser@127.0.0.1                       # wrong password, three times

# On the Linux target: confirm the audit rule is loaded and searchable
sudo auditctl -l
sudo augenrules --load
sudo ausearch -k identity -ts today | tail

# On the sensor host: validate the configuration before trusting it
sudo suricata-update
sudo suricata -T -c /etc/suricata/suricata.yaml -i eth1
sudo tail -f /var/log/suricata/eve.json

# /etc/audit/rules.d/lab.rules — what changed on this host, and who ran what
-w /etc/passwd -p wa -k identity
-w /etc/sudoers -p wa -k privilege
-a always,exit -F arch=b64 -S execve -F key=exec

A Linux audit rule set is inert until it is loaded and it is useless until it is searched, so every rule ends with the same two actions: augenrules --load, then ausearch -k <key> to prove events land. On Windows the equivalent is audit policy plus Sysmon with a maintained community configuration (process creation with command lines, network connections, file creation, registry changes), shipped to the manager by the agent. Then fail a logon there too and find the failure in the interface.

[!TIP] Get one host producing events you can find by name before adding the fifth component. Nineteen parts installed and no working query is the normal way this project dies.

Generating events worth detecting

A detection you have never seen fire is a rumour. You need a controlled source of activity, and there are two established kinds:

A community library of small, documented, technique-mapped tests — individual, reproducible actions mapped to adversary techniques, each with an execution and a cleanup step. Atomic Red Team is the reference example, and it is the one the Start Here page's lab section names.
A framework for running controlled adversary simulations — a simulation platform that chains behaviours into a scenario, well suited to testing detection across multiple hosts. MITRE Caldera is the reference example.

The lesson's rule is absolute: both are run against your own lab targets only, never against anything else, and always with a snapshot taken first.

# Atomic Red Team, on a lab Windows target only, after a snapshot
Import-Module .\invoke-atomicredteam\Invoke-AtomicRedTeam.psd1
Invoke-AtomicTest T1136.001 -ShowDetails     # what this test will actually do
Invoke-AtomicTest T1136.001                  # run it: create a local account
Invoke-AtomicTest T1136.001 -Cleanup         # undo it, then revert the snapshot anyway

Read -ShowDetails before you run anything. The habit of reading the test that dumps credentials or writes a service before executing it is the same habit that keeps you legal everywhere else.

Baseline first: leave the range running for a day with no hostile activity at all, record what normal looks like — login patterns, update traffic, resolver queries, the noise your own backups make — and only then introduce a technique. Detecting a change requires having known the before.

The first five detections

Detection	What it looks for	Data source	Benign cause	Malicious cause
New privileged account	a local or directory account created, or an existing account added to an administrative group	Windows 4720 and 4728; `ausearch -k identity` and `useradd` traces on Linux	a new starter, an automation account provisioned by a script	persistence: an account the attacker keeps
MFA disabled or reset for a user	a factor removed, reset, or an authentication method changed	cloud or identity provider audit records	a legitimate device replacement, a helpdesk reset	the attacker removing the control that would stop them
New service, scheduled task or autorun entry	a new unit file, task, registry run entry or startup item appearing	Windows 7045 and 4698, Sysmon registry and file events, systemd and cron paths	software installation or an update	how nearly all persistence arrives
Successful authentication from a new or unusual source	a logon that is valid but from a source, hour or host not seen before for that account	Windows 4624 with logon type and source address; `Accepted password ... from` in the Linux auth log	travel, a new VPN exit, a second office	valid accounts (root of most intrusions)
Burst of outbound connections to a previously unseen destination	many new outbound destinations, or one repeated at a regular interval	Zeek `conn.log`, Suricata events, firewall egress logs	a new software update channel, a CDN	beaconing, or a first-stage download

Write these five before you write thirty. Each one teaches the full loop: find the field, force the event, tune out the benign cause, and record both outcomes.

Rule discipline and the write-up

A rule earns trust in a fixed order, and skipping a step is how a team ends up with noisy detections nobody reads.

Write the rule with a reason in the description: what behaviour, which adversary technique, why this matters here.
Force the positive. Run the technique against the lab target and confirm the alert fires, with its timestamp, asset and account.
Force the negative. Do something adjacent but benign and confirm the rule does not fire loudly — or fires at a low severity with an explanation you can stand behind.
Record both outcomes in the rule's notes or your register. A rule with a positive and a negative result is a rule you can defend; one without is a guess.

The one-page write-up template every alert should end with:

What fired:            <rule name and severity>
On which asset:        <hostname, address, platform>
For which user:        <account, and how it authenticated>
What it means:         <one sentence a non-specialist would understand>
What I investigated:   <queries run, hosts checked, evidence reviewed>
What I concluded:      <true positive, false positive, or inconclusive>
What I changed:        <rule tuned, control added, gap logged, risk accepted>
ATT&CK mapping:        <tactic and technique, from Lesson 7>

Then the rehearsal routine, which is the whole lesson in five words: snapshot, attack, detect, investigate, revert. Run it until it is boring. When someone asks whether you can detect an attack, you will not describe a product — you will describe this loop, and the results of having run it.

[!WARNING] Never tune a rule by disabling it quietly. Either narrow it, move it to log-only, or document why it is accepted — an unrecorded silent rule becomes a detection gap that nobody knows exists.

Attack it / Defend it

The attack	How it works	The control that stops it
New local administrator account	create an account and add it to an administrative group	alert on 4720/4728 and on `useradd`; audit rules over the account databases
Scheduled task or service for persistence	register a process to run at boot under a plausible name	4698, 7045, Sysmon registry events, and a scheduled review of autoruns
MFA reset to keep access	the attacker removes the second factor on an account they control enough to change	identity-provider audit alerts, changes routed through a helpdesk process with verification
Valid account from a new location	log in with stolen credentials or a session token	4624 with unusual source, impossible-travel logic, conditional access, and honeytoken accounts
Credential dumping on a host	read process memory or the account database after gaining local admin	Sysmon process-access alerting, credential protection, and not letting admin credentials be on that host
Beaconing to command and control	regular, low-volume outbound connections to one destination	Zeek records plus egress logging, and a rule for new destinations and repeated intervals
Covering tracks by clearing logs	clear the event log or the audit trail after activity	alert on log clearing (Windows 1102) and on gaps in collection, which the manager will show as silence
Deleting the evidence of the test	cleanup steps remove artefacts, including the ones your rule needed	record the technique and its cleanup in the drill log so the study is reproducible either way

Key takeaways

The blue-team product is a loop — collect, detect, triage, change. A range that only collects is storage, and a rule nobody tested is a rumour.
Prove each component with a deliberate failure: fail an authentication, fail a logon, write a file, look it up in the interface. Green dashboards prove nothing.
Baseline before you attack. One day of normal activity recorded first is what makes a diff meaningful.
A rule is trusted only after a forced positive and a forced negative, both recorded, along with the reason the rule exists.
Snapshot, attack, detect, investigate, revert is the rehearsal that turns installed software into a skill you can demonstrate.

Check yourself

Name the four stages of the blue-team loop, and say what is missing from a lab that has a working dashboard but never changes a rule.
You have enrolled an agent and the interface shows nothing. List, in order, the three things you would check.
Give the benign and the malicious cause for "a new service appeared on a host", and explain why the rule needs both written down.
Why does the honeypot produce a cleaner alert than a signature-based sensor, and what placement does that depend on?
Write the one-sentence "what it means" line for an alert about a successful authentication from a new country for an account that has MFA enabled but did not prompt.