22 — Logging and Monitoring Foundations
Level: Intermediate · Time: ~18 min · Prerequisites: Lesson 21 — Data Protection and Backups
Why this matters
Detection is impossible without records. Every question an analyst will ever ask during an incident — when did this start, which account was used, what did the process do next, where did it connect — is answered by a log that either exists or does not. The uncomfortable truth about small organisations is that they rarely fail at analysis. They fail at collection: the logs were local-only, or the retention was seven days, or the firewall was never configured to send anything anywhere. This lesson is deliberately unglamorous, because it is the foundation the two lessons that follow stand on, and a SIEM with poor input data is an expensive way of confirming that you know nothing.
The mental model: detection is a pipeline, and it breaks at the weakest stage
collect ship store query alert
──────── ───── ──────── ──────── ────────
host, network, agent, syslog, central index, search, a rule that
identity, cloud, API pull retained long correlate notifies a human
application logs enough
▲ ▲ ▲ ▲ ▲
most failures ────────┴──────────────────┴─────────────────┴────────────────┘
start here
Read it right to left and the commercial pitch appears: buy the right-hand boxes and detection happens. Read it left to right and the reality appears: the right-hand boxes are worthless if the left-hand box is empty. Every hour spent making sure logs arrive, are timestamped correctly and are retained is worth more than an hour spent writing rules.
What to log, by source
| Source | Where it comes from | The items that actually matter |
|---|---|---|
| Endpoints | the endpoint agent, the operating system's own event log, auditd | process creation with command line, parent process, authentication, service and scheduled-task creation, privilege use, USB insertion, AV/EDR detections |
| Network | firewall, DNS resolver, VPN concentrator, DHCP server, IDS | DNS queries, firewall allow and deny, VPN logins, DHCP leases, flow data, IDS alerts |
| Identity and directory | the directory, the identity provider, the SSO platform | logins and failures, MFA events, password resets, group and role changes, directory replication |
| Cloud | the provider's audit trail | API and control-plane operations, IAM changes, storage access, console logins |
| Applications | each business application | authentication, authorisation failures, administrative actions, exports |
The concrete starting set on Windows
| Event ID | Meaning | Why it is in the first ten |
|---|---|---|
| 4624 | successful logon | the baseline of every timeline; logon type and source tell the story |
| 4625 | failed logon | spraying and brute force, visible only in aggregate |
| 4648 | logon with explicit credentials | a process running as another user — lateral movement and admin tooling |
| 4672 | special privileges assigned to a new logon | tells you which sessions are effectively administrative |
| 4720 | a user account was created | persistence, or a fabricated identity |
| 4728 | a member was added to a global security group | how new administrators usually appear |
| 4697 | a service was installed | a common persistence mechanism |
| 7045 | a service was installed (System log) | the same event from the other log — collect both, they do not always agree |
Enable the process-creation telemetry that gives you command lines. On Windows that means Sysmon (free, from Microsoft) rather than only the default audit policy, because the default does not record what was executed with which arguments. Audit policy is configured per category; on a domain, through Group Policy rather than by hand on each machine.
The Linux equivalents
| What | Where it lives | The items to care about |
|---|---|---|
| SSH authentication | /var/log/auth.log on Debian-family, /var/log/secure on RHEL-family, journalctl -u sshd |
accepted and failed logins, source address, key versus password |
| Privilege use | sudo entries in the same files |
who escalated, to what, and which command |
| Scheduled work | journalctl -u cron or /var/log/cron |
a new cron entry is a persistence mechanism |
| Service changes | journalctl from systemd |
unit files created or modified |
| Deep host visibility | auditd with an exec rule in /etc/audit/rules.d/ |
process execution with arguments, file access, privilege changes |
Collect the service and task creation events on servers as well as workstations. Persistence arrives there, and it is quieter than anything a user would notice.
Why command lines and DNS punch above their weight
Of everything in the tables above, two items repay collection more than all the others combined.
Command lines. One field frequently contains the entire incident: the encoded PowerShell argument, the certutil download, the service installed from a temporary folder, the database dump command with the attacker's filename. Without command lines you know that something ran; with them you know what it did and often where it sent the result.
DNS queries. Almost every intrusion needs to resolve a name at some point — for command and control, for download, for exfiltration. DNS logs are small, they are generated for everything, and the query itself shows intent. A workstation querying a domain that has existed for three days is one of the highest-signal single events available, and it is available to any organisation willing to log its resolver. Lesson 44 covers the analysis; here the point is that the collection has to exist first.
The mechanics that decide whether logs are usable
| Decision | The practical answer | The failure if you get it wrong |
|---|---|---|
| Log level | informational and above for security events; debug off in production | debug logging in production is both a flood and a leak: it can write credentials and personal data into a file with weaker protection than the data deserves |
| Collection method | an agent where you need rich host data, syslog where the device is simple, an API pull for cloud services | a device that only supports syslog and is left uncollected |
| Centralisation | ship everything to a remote collector | local-only logs die with the machine the attacker controls, and are the first thing deleted |
| Time | synchronise every device with NTP before you need correlation | see below |
| Retention | long enough to cover the dwell time you actually face | three days of logs answers no question asked a week later |
| Integrity | append-only, remote collector, write-once storage, hashing | an attacker edits history and you cannot tell |
Time synchronisation is the cheapest fix in this module
When clocks disagree, events from different sources cannot be ordered. A login on the identity provider and the process that followed it on an endpoint arrive out of sequence, a firewall deny appears before the request that caused it, and your timeline becomes fiction. Analysts then lose hours arguing about which log is right. The fix costs nothing: point everything at a central time source (chrony, systemd-timesyncd or w32tm on Windows, and check with timedatectl or w32tm /query /status), keep NTP reachable from every segment, and alert when a host drifts. On a Windows domain, member machines take time from the domain hierarchy — so verify the source itself is accurate. Clock skew also breaks TLS certificate validation, which is a second reason to treat it as a defect and not a detail.
Retention arithmetic
Do the estimate rather than guessing. A busy Windows endpoint with process-level telemetry can produce tens to a couple of hundred megabytes of security-relevant log per day. A hundred endpoints therefore means single-digit gigabytes per day before compression — and text logs typically compress well, often by a factor of five to ten. Ninety days across a small estate therefore lands in the hundreds of gigabytes, not the terabytes. Write down the number for your own environment, decide the retention period, and treat storage as a budget line rather than an obstacle.
[!IMPORTANT] Set the retention period from the dwell time you are defending against, not from what fits on the current disk. Intrusions are frequently discovered months after they began, which is why thirty days is often the minimum useful figure and ninety days is a reasonable target for a small organisation.
Log integrity, so history cannot be rewritten
An attacker with administrative rights on the machine will try to clear the log — that is exactly what the event-log-cleared event exists to record. Defence in depth for logs looks like this: ship off the host immediately; store on a collector that the source has no write access to; make the store append-only or write-once where the platform supports it; restrict who can delete or modify; and hash or seal log files so that tampering is detectable. Attackers edit logs precisely because the logs are evidence, and later forensics (Lesson 25) depends on them being trustworthy.
Privacy
Logs contain personal data. User names, source addresses, mailbox subjects and URLs are all personal data under EU rules, so an operational logging programme needs three things documented: the purpose (security monitoring and incident response), the access control (only the people who need it, with their own access logged), and the retention limit. Say this once and design accordingly — the goal is not to make logging legally complicated, it is to avoid building an unmanaged personal-data archive by accident.
Making it real: a minimum viable set
For a small business or a home lab, six sources and ten events is enough to be genuinely useful.
| # | Source | Collect |
|---|---|---|
| 1 | Endpoints | process creation with command line, plus the authentication events |
| 2 | Directory / identity provider | logins, MFA changes, group and role changes |
| 3 | Firewall | allow and deny at the perimeter, particularly outbound |
| 4 | DNS resolver | queries, with the client address |
| 5 | Servers | authentication, service and scheduled-task creation, privilege use |
| 6 | Cloud / SaaS | control-plane and IAM audit trail |
The first ten events to alert on — before you build anything more elaborate — are: a new privileged account or group membership; MFA disabled or re-enrolled; a successful login from a new country; a new service, scheduled task or autorun entry; a login on an account that has been dormant; a spike of failed logins across multiple accounts; log cleared or the audit service stopped; a new local administrator on an endpoint; an outbound connection burst to a new domain; and the security agent going silent. Lesson 23 turns these into rules.
Prove it works: the deliberate-failure test
Do not trust a dashboard; make an event happen and find it.
# 1. Fail a login on purpose on a Linux host, then find your own failure
ssh wronguser@localhost # expect a failure, note the time
# 2. Find it locally
sudo grep -i "failed password" /var/log/auth.log # Debian-family
sudo journalctl -u sshd --since "5 minutes ago" # systemd hosts
# 3. Find it centrally — the part that actually matters
# Search the collector for the same host and the same time window.
# If it is there locally and not centrally, your shipping is broken.
Repeat the pattern for the sources that matter: create a test account and confirm the event arrives centrally; trigger a firewall deny and look for it; sign in from a phone on mobile data and check the country field. Five minutes of deliberate failure teaches you more about your own pipeline than a week of reading vendor documentation.
Three warning signs that logging is broken
| Sign | What it usually means |
|---|---|
| A source has been silent for days | the agent died, the credential expired, or the disk filled and nobody noticed — silence looks identical to "nothing happened" |
| Clocks are out of sync between sources | NTP is not reachable from that segment, and your timelines are already unreliable |
| An alert references events you cannot find | the rule fires on data that is not in your searchable store, so nobody can triage or investigate it |
Treat silence as an alert condition in its own right. A "no events from this source in 24 hours" rule catches more real problems than most detection content ever will.
Attack it / Defend it
| The attack | How it works | The control that stops it |
|---|---|---|
| Log clearing | deletes the local event log after gaining administrative access | remote shipping to a collector the source cannot write to, alerting on the clear event |
| Log tampering | edits entries to remove traces of activity | append-only storage, write-once media, hashing, restricted delete permissions |
| Retention manipulation | waits out a short retention window before acting | retention sized to the dwell time you defend against |
| Clock manipulation | changes host time so correlation fails | central NTP, alert on drift, non-admin time configuration |
| Silencing a source | stops or removes the agent, disables audit policy | agent health monitoring, alert on collector silence, tamper protection on the endpoint agent |
| Log flooding | generates volume to bury the relevant events | rate limits, storage alerts, prioritising high-signal rules over volume |
| Debug-level leakage | verbose logging writes credentials or personal data to disk | log level policy, redaction, protecting and expiring log files |
| Unlogged traffic paths | uses a segment or protocol nobody collects | collect at the perimeter and the resolver, where traffic must pass |
| Collection gap by design | the device only supports syslog and is left uncollected | an asset list checked against sources actually arriving |
Key takeaways
- Detection is a collection problem long before it is an analysis problem. Most small organisations fail at the left of the pipeline, not the right.
- Command lines and DNS give the most value per byte collected. One line can be the whole incident; one query can be the whole intent.
- Nothing can be correlated without synchronised clocks. NTP is free, unglamorous, and the cheapest reliability improvement available.
- Local-only logs are the attacker's first casualty. If a source cannot be reached remotely, its history disappears with the machine.
- Silence is a finding. A source that has stopped sending looks exactly like a quiet network, so alert on the absence.
- Logs are personal data. Purpose, access control and a retention limit — decided in advance, not improvised afterwards.
Check yourself
- An analyst asks what a host was doing last Tuesday and your retention is seven days. What is the actual gap, and what would you change first?
- Why does a two-minute clock difference between the firewall and the domain controller destroy a timeline?
- Which two fields would you insist on collecting before writing any detection rules, and why?
- Name three signs that your logging pipeline is broken even though the dashboard shows no alarms.
- Logs contain user names and source addresses. What three things must be documented about that data?