32 — Host Security and Audit Tools
Level: Intermediate · Time: ~22 min · Prerequisites: Lesson 31 — Traffic Analysis and IDS Tools
Why this matters
Network tools watch the wire. Every host is also its own evidence source: what it is configured to do, what it actually did, and whether anything known-bad is sitting on it. Three different questions, three different tool families, and confusing them is how teams end up with an agent installed that nobody reads. This lesson is about the host half of visibility: auditing a configuration, recording privileged activity, and — the one that changes how a small team works — asking the same question of every machine at once.
The mental model: three questions about a host
| The question | The tool | What it gives you |
|---|---|---|
| "Is this host configured safely?" | a hardening auditor (Lynis) | a score, a list of suggestions, and a trend you can improve |
| "Does it meet a published benchmark?" | a policy scanner (OpenSCAP) | per-rule pass/fail against a machine-readable profile |
| "What did this host actually do?" | kernel audit (auditd) and Sysmon |
records of privileged commands, logins, file writes and process starts |
| "Has anything changed underneath me?" | file integrity monitoring (AIDE) | a diff against a baseline you chose |
| "Is something known-bad here?" | rootkit and malware checkers, YARA | a sanity check against known-bad names, hashes and patterns |
| "What exists out there?" | the endpoint query engine (osquery) | inventory and ad-hoc questions across the whole estate |
| "Something happened — collect everything, everywhere." | triage collection at scale (Velociraptor) | artefacts from many machines in minutes, centrally |
None of these replaces another. Read the table as a division of responsibility, not a shopping list: a small team can run rows two, four, six and seven well, and be in better shape than most organisations with all seven installed and ignored.
Auditing configuration and hardening guidance
An automated auditor reads your machine's configuration and compares it against a body of hardening knowledge, then reports. Lynis is the one to start with, because it needs no policy document, no server and no agent — it is one command on one host.
sudo lynis audit system --quick # full audit, fewer prompts
sudo lynis show groups # the test categories it uses
sudo lynis show details SSH-7408 # what a specific suggestion meant here
sudo grep -E 'hardening_index|tests_performed' /var/log/lynis-report.dat
How to read it: the output groups tests by area (boot, kernel, authentication, networking, services, SSH, logging), gives each a result, and finishes with a hardening index — a number out of 100 that is useful only as a per-host trend. Each finding carries an identifier such as SSH-7408, the raw evidence it saw, and a suggestion. Two rules for interpreting it honestly:
- Most suggestions are judgement calls. A host with no SELinux on a supported distribution is a real finding; a warning about a package you deliberately installed is a conversation, not a defect. You are not trying to reach 100.
- Scores are not comparable between hosts. A mail relay, a database server and a laptop with a different role will never match. Compare a host to its own previous score, after a change you made on purpose.
The workflow is the point: run it, fix what matters for that host's role, run it again, and write down the new score. That record is the evidence that your baseline is improving rather than merely asserted.
Where you need to demonstrate compliance against a published benchmark, an automated scanner evaluates a machine-readable policy profile and produces per-rule results:
oscap info /usr/share/xml/scap/ssg/content/ssg-ubuntu2204-ds.xml # list available profiles
sudo oscap xccdf eval \
--profile xccdf_org.ssgproject.content_profile_cis_level1_server \
--results /root/scan-results.xml --report /root/scan-report.html \
/usr/share/xml/scap/ssg/content/ssg-ubuntu2204-ds.xml
Each rule returns pass, fail, not applicable or not checked, with its rationale and remediation text. The honest caveat: a failed check is the beginning of a conversation. Some rules document a decision you have not made yet, some conflict with the reason the host exists, and the scanner has no idea what the business needs it to do. Automated remediation (--remediate) exists — review what it will change before running it on anything you care about.
[!TIP] Run the auditor on a scratch copy of your standard build before it touches production. Fixing findings there, then shipping that image, is the difference between a baseline and an archaeology project.
Watching a host: the kernel audit subsystem
Configuration tells you intent. Audit tells you events. On Linux the kernel audit subsystem records security-relevant activity and can be scoped precisely with rules.
# Identity and privilege files: anything written or with permissions changed
-w /etc/passwd -p wa -k identity
-w /etc/sudoers -p wa -k identity
# Only root-initiated command execution — the volume-friendly version
-a always,exit -F arch=b64 -S execve -F euid=0 -k exec_root
# Kernel module loading and a web root
-a always,exit -F arch=b64 -S init_module,finit_module -k modules
-w /var/www/html -p wa -k webroot
Read the syntax as: -w watches a path, -p wa fires on writes and attribute changes, -a always,exit appends a rule to the exit filter, -F arch=b64 restricts to 64-bit syscalls, -S execve names the syscall, and -k attaches a key — the label that groups related events so you can find them later. Rules live in /etc/audit/rules.d/50-local.rules; load them with augenrules --load and confirm with auditctl -l. Then search what was recorded:
sudo ausearch -k identity -i # -i resolves UIDs and syscall names
sudo ausearch -k exec_root -ts today -i | head
sudo aureport -k --summary # events per key, which is your tuning report
sudo aureport --auth --summary
[!WARNING] Auditing every
execveon a busy server generates enormous volume and will fill a disk. Scope to what answers a question —euid=0for privileged execution, one key per question — and checkaureport -kafter a week to see whether each rule earned its keep.
Two operational notes. Adding -e 2 at the end of your rules makes the audit configuration immutable until reboot, which stops an intruder quietly removing your visibility — it also means every future change needs a reboot, so plan it. And record why each rule exists in a comment, because a ruleset nobody understands gets disabled the first time it fills a disk.
File integrity monitoring and malware checks
File integrity monitoring answers a question no log can: has this file changed since I decided it was correct? You build a baseline, then you are told about differences.
sudo aideinit # build the baseline database
sudo cp /var/lib/aide/aide.db.new /var/lib/aide/aide.db
sudo aide --check # compare against the baseline
On Debian-family systems that first command is the aideinit wrapper; upstream AIDE uses aide --init and the same database path. Selection lines in /etc/aide/aide.conf decide what is watched and what is compared — /etc Full, /boot Full, /var/www Full — with Full being a shorthand for the attributes to monitor (permissions, inode, links, owner, group, size, timestamps, checksums). The operational reality is unavoidable: package updates cause legitimate changes, so every diff needs an explanation, and a checker that cries wolf every patch Tuesday is one you will stop reading. The rule that makes it worth the effort is that an unexplained change to a system binary, an SSH configuration or a web root is one of the strongest signals available to you — it is the point where "something happened" becomes "someone is here".
Rootkit and malware checkers are the other half of the sanity check:
sudo rkhunter --update && sudo rkhunter --check --skip-keypress
sudo chkrootkit
sudo freshclam && sudo clamscan -r -i /home /srv # -i = only report infected files
Their honest limit: rootkit checkers look for known-bad names, expected hashes and simple anomalies, and competent attacker tooling looks like the legitimate software it replaced. They produce false positives on ordinary software and miss anything written for the purpose. Treat them as a smoke alarm, not an assurance. Where they genuinely earn their place is scanning content rather than the operating system: mail gateways, file shares, removable media and shared folders, where signature matching against known malware is exactly the right tool. Signature scanners can also be run on access by a daemon, which is what makes a file share worth protecting on the way in and out. For your own pattern matching across files and memory, a rule-driven scanner such as YARA does the job:
Asking questions of the whole estate: the endpoint query engine
The single most useful tool in this lesson for a small team is an endpoint query engine that exposes the machine as a set of queryable tables. You write a question in SQL-like form, and it answers from live system state. On one host it is a convenience; deployed to every host with a scheduled fleet, it is an inventory you can interrogate.
osqueryi # interactive shell on one machine
osqueryi "SELECT pid, name, path FROM processes WHERE path LIKE '/dev/shm/%';"
osqueryi --json "SELECT name, version FROM deb_packages WHERE name = 'openssl';" > openssl.json
Real questions, with the table that answers each:
| Question | Query |
|---|---|
| Which machines run this software version? | SELECT name, version FROM deb_packages WHERE name = 'openssl'; (rpm_packages on RHEL-family, programs on Windows) |
| Which hosts have something listening on this port? | SELECT p.name, p.path, l.port FROM listening_ports l JOIN processes p USING (pid) WHERE l.protocol = 6 AND l.port != 0; |
| Which users can log in interactively? | SELECT username, shell FROM users WHERE shell NOT LIKE '%nologin%' AND shell != '/bin/false'; |
| Which machines are unpatched? | SELECT hotfix_id, installed_on FROM patches; on Windows, plus SELECT * FROM os_version; and the package table |
| What runs automatically here? | SELECT name, path, status FROM startup_items; |
| Has anything run from a temporary directory? | SELECT pid, name, path FROM processes WHERE path LIKE '/tmp/%'; |
| What changed in a watched directory? | SELECT * FROM file_events; (requires the file-monitoring configuration) |
Run osqueryd as a service with a config file to schedule those queries and log results centrally, and keep the packs and queries in version control with the rest of your configuration. Two habits make the difference between a toy and an asset: keep queries that produced a useful answer (they become your inventory questions), and treat a surprising result as a lead rather than a conclusion.
Triage at scale, the Windows tools, and the weekly routine
Older incident response meant visiting each machine. The modern version asks all of them at once. Velociraptor is the open-source tool for that: a server with lightweight clients, a query language, a library of packaged artefact collections, and hunts that run one collection across every enrolled client and return the results to the server. "Did this file exist anywhere else?" or "which machines have this persistence entry?" changes from a two-day walk-around to a query. It complements rather than duplicates the endpoint query engine: one answers standing questions continuously, the other collects deep artefacts on demand for a specific investigation.
On Windows, a handful of tools cover most of the ground, and all of them are free:
| Tool | What it shows |
|---|---|
| Sysmon | process creation with command lines and hashes, network connections, image loads, registry changes — written to the event log so your central platform sees it |
| Process Explorer | the live process tree, handles, loaded modules, and signature status of what is running |
| Process Monitor | real-time file, registry, network and process activity for a window you reproduce yourself |
| Autoruns | every persistence location on the machine, with signature verification next to each entry |
| TCPView | live TCP and UDP endpoints and the process that owns each one |
| sigcheck | version, signature and timestamp for a file — how you spot unsigned or oddly signed binaries |
Get-WinEvent -FilterHashtable @{LogName='Security'; Id=4624,4625} -MaxEvents 50
wevtutil qe Security /c:20 /rd:true /f:text
Get-LocalGroupMember -Group Administrators
Get-NetTCPConnection -State Listen | Sort-Object LocalPort
Turn on PowerShell script block logging (event 4104) and module logging through Group Policy as well. Without them, the command an intruder typed — often the most useful single artefact in an investigation — is not recorded anywhere.
A weekly host check for a small environment, in the form of one table:
| Check | How |
|---|---|
| Anything new listening? | sudo ss -tulpn, and compare against last week's list |
| Accounts and admin groups changed? | sudo getent passwd, sudo getent group sudo, Get-LocalGroupMember -Group Administrators |
| Authentication failures and privileged commands | sudo aureport --auth --summary and sudo aureport -k |
| Integrity diff | sudo aide --check |
| Patch and reboot state | kernel version and package upgrade history; SELECT * FROM os_version; |
| Malware and rootkit smoke test | sudo chkrootkit and sudo clamscan -r -i /srv |
Finally, the baseline. Pick one reference machine per role. Run the auditor on it. Fix what genuinely matters for that role. Record the expected state — the audit report, the integrity database, an inventory snapshot of accounts, ports and startup items, kept in version control. Then treat deviation from that record as the signal, and re-baseline deliberately when you change something. Detection is only ever a comparison against a stated expectation; without the expectation you have data, not monitoring.
Attack it / Defend it
| The attack | How it works | The control that stops it |
|---|---|---|
| Persistence via a new service or scheduled job | register something to run at boot under a plausible name | startup-item inventory, scheduled queries for new services, Sysmon registry events |
| Local account creation for access | add a user or elevate one into the admin group | audit rules on /etc/passwd and /etc/group, alerting on group membership change, review with aureport |
| Binary replacement (trojanised tool) | swap a system binary for one that does the same job plus something else | file integrity monitoring on system paths, and an explained diff after every update |
| Rootkit installation | conceal processes and files from the operator | immutable audit configuration, boot-time integrity checks, kernel module load auditing |
| Living off the land | use the tools already on the machine, so nothing is installed | command-line visibility: script block logging, Sysmon process creation, audit records for privileged execution |
| Configuration drift | a server quietly widens its exposure over months | scheduled audits against the baseline, with the score recorded per host |
| Insider misuse of a host | a legitimate account reads or exports what it should not | activity records with attribution, plus access reviews of who holds local admin |
| Malware arriving on shared files | a user opens something that came off a share or a USB stick | signature scanning on the file server and on access, plus egress control |
Key takeaways
- Configuration audit, activity audit and malware scanning answer three different questions. The mistake is installing one and believing it covers the other two.
- Hardening scores are trends, not grades. Fix what matters for the host's role, record the number, and compare a machine only to its own past.
- Audit rules cost disk and attention. Scope to what answers a question, use keys to group events, and report on volume after a week.
- File integrity monitoring is only as good as the discipline behind it. Updates cause legitimate changes; the value is in the changes nobody can explain.
- The endpoint query engine is the highest-leverage tool here for a small team. It turns "log into each server" into "ask all of them", and the questions you keep re-asking become your inventory.
Check yourself
- You need to know whether any machine in a 200-host estate still runs a particular vulnerable library version. Which tool do you use, and what would you have to have configured beforehand?
- Your auditor reports a failed benchmark rule that the host's owner says is intentional. What do you record, and why is the failure still useful?
- Write an audit rule that records every command run as root, and explain why the same rule without
euid=0is a bad idea on a busy server. - A new kernel package was installed last night and your integrity checker reports 40 changed files. What do you do, and what would make you suspicious anyway?
- Which single Windows setting would you enable first on an estate, and what does it record that nothing else does?