35 — SIEM, Logging and Monitoring Stacks
Level: Advanced · Time: ~22 min · Prerequisites: Lesson 34 — Network Defence Tools
Why this matters
Lesson 22 — Logging and Monitoring Foundations said what to log and why; Lesson 23 turned observations into detections. This lesson is the software that makes both real: collection, storage, search, rules and dashboards running on a machine you operate. The blunt advice up front — a small team should start with one integrated platform rather than assembling five — is not a preference. Every component you add needs patching, backup, upgrades, tuning and someone who understands it, and a stack nobody owns produces alerts nobody reads.
The mental model: one platform before five
The question is not "which is the best SIEM". It is "how many moving parts can this team genuinely maintain in a year, at 03:00, with one person on call". A single integrated platform covers the whole path from agent to alert; assembling the parts yourself means owning each joint.
| The job | One integrated platform | Assembling it yourself |
|---|---|---|
| Collecting logs from hosts | an agent per operating system, with enrolment | a shipper per platform, plus certificates and config management |
| Decoding and normalising | built-in decoders for common sources | writing and maintaining a parser per log format |
| Detection rules | shipped rule set plus local rules | authoring rules in your own language, with your own test harness |
| Dashboards | default views per module | building and maintaining every visualisation |
| File integrity, inventory, vulnerability data | modules included in the agent | separate tools, separate tuning, separate upgrade paths |
| Automated response | built-in actions on a rule match | one more integration to build, and to fail |
Start with the integrated platform, run it for six months, and let your actual gaps tell you whether you need the heavyweight stack. Most small environments never do.
The recommended starting point: agent plus manager
The platform that fits this description best is Wazuh, and it is worth understanding its four parts, because everything you learn transfers to any other SIEM.
| Part | What it does |
|---|---|
| Agent | runs on each host: collects log files, watches files for changes, inventories packages and configuration, and ships it all to the manager |
| Manager (server) | receives events, runs decoders to parse them and rules to judge them, and triggers response actions |
| Indexer | stores and indexes alerts and events so they can be searched at speed |
| Dashboard | the web interface: alerts, dashboards, and the search box you will live in |
Enrolment is the part people get wrong, so do it deliberately on one host first:
# On the agent host — register with the manager and start shipping
sudo /var/ossec/bin/agent-auth -m 10.20.30.10
sudo systemctl restart wazuh-agent
# On the manager — the management tools live in /var/ossec/bin/
sudo /var/ossec/bin/agent_control -l # what has enrolled, and its last keepalive
Rules and decoders are XML, and local ones belong in /var/ossec/etc/rules/local_rules.xml and /var/ossec/etc/decoders/local_decoder.xml so an upgrade does not overwrite your work:
<group name="local,sshd,">
<rule id="100100" level="10" frequency="8" timeframe="120">
<if_matched_sid>5716</if_matched_sid>
<same_source_ip />
<description>Repeated SSH failures from one address, then a success</description>
<mitre>
<id>T1110</id>
</mitre>
</rule>
</group>
Rule levels run from 0 to 15, where 0 is ignored and the higher numbers are what reach a human. Two habits make local rules trustworthy: test the rule against real log text before trusting it (the manager ships a log-testing tool, /var/ossec/bin/wazuh-logtest, which shows you the decoder and rule that matched), and verify every rule ID you reference against your own ruleset, because if_matched_sid is meaningless if the ID belongs to someone else's version.
What you get beyond rules: file integrity monitoring (watch the paths that must never change, and schedule the scans), a security configuration assessment that flags hardening drift host by host, and vulnerability detection built from the package inventory. All of it reports — none of it patches, so the platform tells you what to do and you still have to do it.
Automated response deserves its own warning. The platform can act on a rule match — adding a firewall block for a source address, for example — but a rule firing is a judgement you wrote in advance, not a decision made with evidence. Start with notification, then temporary blocks, and always carry an exclusion list for your own infrastructure and monitoring.
[!IMPORTANT] Default rules are broad by design, and they will be noisy in your environment. Run the same discipline as the IDS in Lesson 31: disable what cannot apply to you, record why each rule was changed, and re-tune after every upgrade — because a platform nobody trusts becomes a platform nobody opens.
The heavyweight stack, and the arithmetic of log volume
The other established shape is assembled from three layers, each of which you run and scale separately.
| Layer | Common choices | What it does |
|---|---|---|
| Search and analytics engine | Elasticsearch or OpenSearch | stores, indexes and queries everything |
| Collection and shipping | the Beats family (Filebeat, Winlogbeat, Auditbeat), Logstash, or a lightweight forwarder such as Fluent Bit or Vector |
reads logs and events, adds fields, delivers them |
| Web interface | Kibana or OpenSearch Dashboards | dashboards, saved searches, the analyst's view |
The storage engine is the expensive part, because it is the only component whose cost grows with everything you collect. Do the arithmetic before you deploy, or the disk will do it for you:
| Source | Typical daily volume |
|---|---|
| A quiet Linux server | 20–80 MB |
| A Windows server with security auditing and an agent | 100–400 MB |
| A workstation with endpoint telemetry enabled | 50–200 MB, and more with verbose process logging |
Fifty hosts averaging 60 MB a day is about 3 GB a day, which is 90 GB over 30 days. Add one replica and index overhead — a factor of about two — and a month of retention is nearer 180 GB; ninety days is half a terabyte before you have answered a single question. Compression and storage tiers (keep recent data on fast disks, older data on cheap ones or in snapshots) recover part of that, so retention is a budget decision, not a technical one. Two guardrails: set retention per source (security events 90 days, debug logs 7 days), and drop the debug logs at the shipper, because nobody reads them and they are most of your bill.
Lighter alternatives, metrics, and all-in-one distributions
Not every environment needs a search cluster. A centralised log server with a web interface and a modest database behind it is the sensible middle — Graylog is the established open-source option — and every environment should have at least a plain syslog collector, so logs survive a host being destroyed:
# /etc/rsyslog.d/50-forward.conf — send everything to the log host over TCP
*.* action(type="omfwd" target="10.20.30.10" port="514" protocol="tcp")
Then there is the distinction that rescues more budgets than any other sentence in this lesson: metrics and logs are different things.
| Metrics | Logs | |
|---|---|---|
| Shape | numbers over time, fixed set of labels | events with detail and arbitrary fields |
| Cost | cheap; a series per minute per host | expensive; grows with verbosity |
| Best at | trends, capacity, availability, "is it normal this week" | investigation, "what exactly happened" |
| The tool | a time-series database (Prometheus) plus a dashboard (Grafana) |
the logging platform above |
| Limits and retention | months, easily | days to weeks without real money |
Collect both, and do not confuse them. Prometheus pulls from exporters — node_exporter is the standard on a Linux host — and you query it with its own language (rate(node_cpu_seconds_total{mode="system"}[5m])), while Grafana draws the picture and can read from both a time-series database and a log store. The classic errors are trying to answer "which process opened this file?" from a graph, and trying to graph a million log lines.
Finally, the pre-packaged security distribution: an all-in-one image that bundles network capture, intrusion detection and an analysis console with a strong default configuration (Security Onion is the reference example, and SELKS is a similar idea wrapped around Suricata). It makes excellent sense on a dedicated analyst machine or in a lab, because someone else has already made a hundred good decisions about how the pieces fit. It makes poor sense for a small business that cannot maintain it — these distributions expect an operator who reads alerts for a living, and an unmaintained one is worse than nothing.
Threat intelligence, cases and automation
Structured threat intelligence turns scattered reports into data you can use. A platform for collecting, storing and sharing indicators (MISP is the open-source standard) gives you events with attributes, taxonomies to classify them, feeds from other installations, sharing groups, and — the part that saves your time — warning lists that suppress indicators which are actually false positives because they belong to a legitimate service. The difference from reading blogs is that intelligence in this form is machine-processable: it can be exported into the detection tools from Lesson 31 as indicator sets, and searched against your own logs.
[!TIP] Import two curated feeds, not twenty. Then check what your own logs show for those indicators. Collecting threat intelligence you never act on is a hobby; acting on two feeds correctly is a control.
Cases and automation are what separate a professional response from an inbox. A case-management platform (TheHive, with Cortex alongside it for automated analysis of an observable such as a file hash or an address) turns a pile of alerts into a record with an owner, tasks and evidence — which is exactly what your post-incident review and your insurer will ask for. Automation ties the boring steps together: enrich this address, look up this hash, open a case, notify the channel.
The warning is worth more than the recommendation: automate after you understand the process, not before. Automating a flow nobody has ever performed manually produces wrong answers faster. Automate enrichment and notification first, keep the decision with a human, and only hand over a decision once the flow has been right a hundred times in a row.
A reference architecture, and the operating reality
For a fifty-host small organisation, in prose: every server and workstation runs an agent that ships to one manager; a single network sensor on a span port sends its connection, DNS and alert records to the same store; the firewall and the DNS resolver forward their logs by syslog; everything lands in one indexer with enough disk for the retention you have actually budgeted; and the whole thing is read through one dashboard, with five rules and one person on the queue for thirty minutes a day.
| Component | What runs | Where it sits |
|---|---|---|
| Host collection | the platform agent | every server and workstation |
| Network collection | the logging engine and IDS records from Lesson 31 | sensor on a span port at the edge, and between zones |
| Infrastructure logs | firewall, resolver, switch and VPN logs by syslog | forwarded to the same store |
| Storage and search | the platform's indexer | one server with sized disks, and a backup |
| Rules | shipped rules plus your five local rules | in version control |
| The read surface | one dashboard, one queue, one named owner | whoever is on duty today |
The five rules worth starting with: repeated authentication failures followed by a success from the same address; a new privileged group member or local administrator; an integrity change to a system binary or a web root; outbound traffic to a destination category you never normally talk to, with unusual volume; and an endpoint protection detection, which is never the whole story on its own.
Two things to plan before you need them. Back up the log platform itself — its configuration, its rules and a tested restore — because the platform is now one of the most privileged systems you own, and it should have its own separate admin credentials and MFA on its web interface. And decide the retention numbers with the person who pays the storage bill, because growth is unbounded otherwise.
The anti-pattern, in one sentence: deploying a large stack, collecting everything, and reading nothing. A platform that produces thousands of unread alerts teaches the whole organisation that alerts are noise, and the next budget review removes it — leaving you with none of the visibility you were trying to buy.
Attack it / Defend it
| The attack | How it works | The control that stops it |
|---|---|---|
| Clearing or tampering with local logs | erase evidence on the host you control | ship logs off the host immediately, restrict who can stop the agent, alert on logging gaps |
| Disabling the audit or logging service | turn off what would record the next step | integrity monitoring of logging configuration, alert on service state changes |
| Log injection | write forged entries that mimic another host or user | trusted time source, structured parsing, alert on anomalies in your own logs |
| Alert flooding | drown the queue so the real signal is missed | thresholds and aggregation, tuning discipline, and a small rule set you trust |
| Attacking the platform itself | the SIEM holds everything, so it is worth owning | separate admin credentials, MFA, patched indexer, network-restricted management interface |
| Evidence expiry | wait until retention has deleted the relevant window | retention decisions made deliberately, with long-term storage for anything required |
| Living on an unmanaged host | operate where no agent runs | inventory, agent-deployment enforcement, network evidence as the fallback |
| Unsafe automated response | a rule blocks your own infrastructure, or the attacker triggers it deliberately | notification before blocking, exclusion lists, rate limits on automated actions |
Key takeaways
- Start with one integrated platform. Five components maintained badly give you less visibility than one maintained well.
- Host collection plus network records plus infrastructure syslog is the whole architecture. Everything else is presentation.
- Storage is the only cost that grows with everything you collect, so retention is a budget decision, and per-source retention is how you keep it affordable.
- Metrics and logs answer different questions. Graphs show trends; logs show events. Collect both, and never confuse them.
Check yourself
- Your manager asks why you cannot just install five best-of-breed tools. Give three maintenance arguments in one sentence each.
- Fifty hosts average 60 MB of logs a day. Roughly how much storage do you need for 90 days with one replica, and which retention decision saves the most?
- You want an alert when an internal host connects to a category of destination it never normally reaches. Which two components from earlier lessons feed that rule?
- A local rule references an event ID that does not exist in your installation. What is the symptom, and how would you have caught it earlier?
- Give one example of automation you should enable on day one, and one you should not.