02 — Protocols You Must Know
Level: Beginner · Time: ~15 min · Prerequisites: Lesson 1 — Networking Fundamentals for Security
Why this matters
Attacks do not target "the network" — they target protocols. Phishing arrives over HTTPS, ransomware spreads over SMB, stolen data leaves through DNS, credentials leak from an unencrypted LDAP bind or a default SNMP community string. Every firewall rule, TLS policy and detection rule you will ever write is a statement about what a protocol is supposed to look like. If you know the protocol, the anomaly is visible; if you do not, everything looks like noise.
The mental model: TCP is a phone call, UDP is a postcard
| TCP | UDP | |
|---|---|---|
| Connection | established first, then used | none — fire and forget |
| Delivery | ordered, acknowledged, retransmitted | best effort; may be lost or reordered |
| State | both ends keep session state | none |
| Spoofing | hard — replies need the right sequence state | easy — the source address can be forged freely |
| Typical use | web, SSH, mail, file transfer, SMB, RDP | DNS, DHCP, NTP, SNMP, VoIP, QUIC/HTTP3, VPNs |
| Attack shape | hijacking, SYN flooding, RST injection | amplification and reflection, blind spoofed scanning |
The choice of transport is a security decision that was already made for you. Anything that needs to survive loss and cannot be spoofed went to TCP. Anything optimised for speed and size went to UDP — and inherited the ability to lie about where it came from.
The three-way handshake
Client Server
|------- SYN, seq=x --------->| I want to talk. My sequence number is x.
|<- SYN+ACK, seq=y, ack=x+1 --| Agreed. Mine is y, and I have your x.
|------- ACK, ack=y+1 ------->| Acknowledged. Session is open.
| Flag | Meaning | Security relevance |
|---|---|---|
| SYN | open a connection, synchronise sequence numbers | half-open SYNs are cheap to send and expensive to keep — SYN floods |
| ACK | this segment acknowledges received bytes | forged transfers off-path are hard without the right numbers |
| PSH | deliver to the application immediately | reveals interactive sessions in a capture |
| URG | urgent pointer — rarely used, historically mishandled | odd URG traffic is unusual enough to be worth a look |
| FIN | orderly close, one direction at a time | graceful teardown |
| RST | abort the connection immediately | forged RSTs kill live sessions: session disruption and censorship |
Sequence numbers, and why sessions can be stolen
TCP identifies a session by the four-tuple — source IP, source port, destination IP, destination port — plus the sequence state agreed at handshake time. Every byte that follows is numbered. That design has two consequences:
- Off-path attackers must guess. A blind attacker has to guess the initial sequence number and the port before the session ends. Modern stacks randomise the initial sequence number precisely to make that impractical — which is why you should never assume an old or embedded stack does.
- On-path attackers need guess nothing. Anyone who can see the traffic (same unencrypted segment, a man-in-the-middle position, a compromised router) knows the four-tuple and the sequence numbers. From there they can inject data into the session or forge a RST and tear it down. This is TCP session hijacking, and the only real defence is encryption above TCP — the attacker cannot forge the TLS records. See Lesson 9 for how this is exploited and Lesson 18 for the controls that stop it.
UDP: connectionless, cheap, and spoofable
UDP has no handshake, so an attacker can send a packet claiming to come from anyone — the victim's address, for instance. If the service replies, the reply goes to the victim. That is reflection, and if the reply is much larger than the request it is amplification. Classic reflectors are DNS resolvers, NTP, SNMP, open memcached instances and SSDP. A 60-byte query producing a kilobyte-scale answer turns one attacker into a distributed denial-of-service with no bots required.
[!WARNING] An open recursive DNS resolver on your network is a weapon aimed at somebody else's. Restrict recursion to your own clients, and disable the old NTP
monlistquery. This is basic internet hygiene, and it is also what BCP 38 ingress filtering — dropping packets with a source address that could not legitimately come from your network — exists to reduce.
The protocol tour
| Protocol | What it does | Default port(s) | Security note |
|---|---|---|---|
| DNS | names to addresses | 53 UDP/TCP (853 DoT, 443 DoH) | unauthenticated by default; the most abused protocol on the internet |
| DHCP | hands out address, gateway, DNS | 67 server / 68 client | no authentication: a rogue server becomes your gateway instantly |
| ARP | maps local IP to MAC | none — link layer | unauthenticated: spoofing and poisoning on the same segment |
| ICMP | errors and diagnostics | none — IP protocol 1 | reconnaissance (echo), tunnelling, redirects |
| HTTP | the web | 80 TCP | cleartext: any hop can read and modify it |
| HTTPS | the web over TLS | 443 TCP | protected, but only as strong as certificate validation |
| TLS | encryption and authentication layer | carried on 443, 993, 636, 587, 25 | what is negotiated matters; old versions and weak ciphers are the risk |
| SSH | remote shell, tunnels, SFTP | 22 TCP | keys beat passwords; internet-facing SSH attracts constant brute force |
| FTP / FTPS / SFTP | file transfer | 21 (+20 data), 990, 22 | plain FTP sends the password and the file in cleartext |
| SMTP / IMAP / POP3 | sending and reading mail | 25, 587, 465 / 143, 993 / 110, 995 | STARTTLS must be required, not offered, or it can be stripped |
| SMB | Windows file sharing, AD, remote admin | 445 TCP, 139 legacy | SMBv1 must die; 445 is the lateral-movement highway |
| RDP | remote desktop | 3389 TCP/UDP | brute force and pre-auth RCE; never expose it directly |
| LDAP | directory queries and binds | 389 (636 LDAPS), 3268/3269 GC | plain and anonymous binds leak both credentials and the directory |
| Kerberos | ticket-based authentication | 88 TCP/UDP | ticketing attacks such as Kerberoasting — Lesson 13 covers them |
| SNMP | monitoring routers, switches, printers | 161 queries, 162 traps | v1/v2c community strings are cleartext passwords; use v3 |
| NTP | clock synchronisation | 123 UDP | wrong time breaks Kerberos, certificate validity and log correlation |
| Telnet | remote shell | 23 TCP | cleartext, including the password. Use SSH. Always. |
DNS: the most-abused protocol
A stub resolver on your machine asks a recursive resolver (your ISP, your router, or 1.1.1.1), which walks the hierarchy — root, then the top-level domain, then the domain's authoritative name servers — and caches the result for the TTL.
| Record | Answers | Worth knowing |
|---|---|---|
| A | the IPv4 address | what a browser needs |
| AAAA | the IPv6 address | same, version 6 |
| CNAME | "this name is really that name" | an alias; a common takeover target when the target is deleted |
| MX | where mail for the domain goes | a target list for phishing infrastructure |
| TXT | arbitrary text | where SPF, DKIM and DMARC live — and where data is hidden |
| NS | which servers are authoritative | delegation; a hijack here owns the domain |
DNS is abused because it is allowed through almost every firewall, it is rarely inspected, and it is rarely centralised. Data exfiltration in TXT or subdomain labels, command-and-control through long random names, and DNS tunnelling all look like ordinary queries to a device that only checks the port number. Encrypted DNS (DoT on 853, DoH on 443) makes the user's traffic private and the network defender's job harder at the same time.
DHCP, ARP and ICMP: the unauthenticated trio
DHCP DORA exchange
Client 68 ──── DISCOVER (broadcast: "is there a DHCP server?") ───► Server 67
Client 68 ◄─── OFFER ("use 192.168.1.57, gateway .1, DNS .1") ── Server 67
Client 68 ──── REQUEST ("I would like .57") ─────────────────────► Server 67
Client 68 ◄─── ACK ("it is yours for 86400 seconds") ──────── Server 67
None of the four steps is authenticated or encrypted, which is why a rogue DHCP server on a café network can hand out itself as the gateway and DNS server. ARP has the same weakness — "who has 192.168.1.1?" is a broadcast anyone may answer, and nothing verifies the reply. ICMP is a legitimate diagnostic tool (echo request/reply is what ping sends, destination-unreachable and time-exceeded carry real routing errors) and simultaneously a channel: ICMP tunnelling tools carry a TCP stream inside echo requests, and ICMP redirects can rewrite a host's routing table if they are trusted.
Mail: STARTTLS and the downgrade window
Port 25 is server-to-server relay, 587 is authenticated client submission, and 465 is implicit TLS. On the reading side, IMAP lives on 143/993 and POP3 on 110/995. STARTTLS upgrades a plaintext connection to TLS — which means an attacker who can sit in the middle can strip the offer and keep the session in cleartext. Require TLS rather than offering it, and treat an unauthenticated inbound message as untrusted input. SPF, DKIM and DMARC (Lesson 43) are the controls that make spoofed senders visible.
Cleartext versus encrypted
| Job | Cleartext | Encrypted equivalent |
|---|---|---|
| Remote shell | Telnet 23 | SSH 22 |
| Web | HTTP 80 | HTTPS 443 |
| File transfer | FTP 21 | SFTP on 22 or FTPS on 990 |
| Mail submission | SMTP 25 | SMTP 587 with required STARTTLS, or 465 |
| Mail reading | IMAP 143 / POP3 110 | IMAP 993 / POP3 995 |
| Directory | LDAP 389 | LDAPS 636 |
| Device monitoring | SNMP v1/v2c | SNMPv3 with authentication and privacy |
| Name resolution | DNS 53 | DoT 853 or DoH 443 |
| Time | NTP 123 | NTP has authenticated variants, but adoption is patchy |
| Windows file sharing | SMB 445 | SMB 445 with SMB3 encryption enabled and required |
If a protocol appears in the left column on a network you are responsible for, that is the finding — before anyone has exploited anything.
Open is not the same as vulnerable
An open port means something is listening. A vulnerable service means that specific version, in that specific configuration, can be made to do something it should not. They are different claims and they need different evidence.
- Port numbers are convention, not enforcement. Anything can listen on 443 — a management web UI, an unpatched appliance, a backdoor. Never assume a service from its port; confirm it.
- Version matters more than port. The same port 445 is a hardened, patched Windows file server or the SMBv1 vulnerability that WannaCry used to spread (MS17-010). The port tells you nothing; the build does.
- Configuration is where most findings live. Default credentials, anonymous binds, unlimited authentication attempts, verbose banners, backup files left in a web root.
- Exposure is a risk multiplier. A vulnerable service reachable only from a management VLAN is a ticket. The same service on the internet is an incident waiting for a scanner to find it — usually within hours.
Commands worth running
Read-only, and on your own machines, lab or the network you own.
# What does this name actually resolve to, and through whom?
dig example.com +short
dig example.com MX
dig +trace example.com # walk root → TLD → authoritative yourself
# What is listening on this host right now?
ss -tulpn # Linux: TCP+UDP, listening, with process (netstat -ano on Windows)
# What is that service really? Port number is a hint, banner is evidence
nmap -sV -p 443,445,3389 192.168.1.0/24
nmap -sU -p 53,123,161 192.168.1.10 # UDP needs its own scan; it is slower
# Is this actually the certificate I expect, and is the chain complete?
openssl s_client -connect example.com:443 -servername example.com
# Is the port reachable at all, from where I am?
nc -vz example.com 443
# Watch the protocol in flight
tcpdump -ni any port 53 -c 20
Reading port 53 traffic from your own network for five minutes will show you which machines are resolving names and whether DNS is going where you think it is.
Attack it / Defend it
| The attack | How it works | The control that stops it |
|---|---|---|
| DNS tunnelling and exfiltration | data encoded in query names or TXT records to an attacker's authoritative server | restrict resolvers, monitor query volume and length, block direct-to-internet DNS, inspect egress |
| DNS cache poisoning | inject a forged answer so a name points at the attacker's IP | DNSSEC validation, a resolver that randomises source ports, trusted forwarders |
| DNS spoofing / pharming | answer a query before the real resolver does, on the local segment | DHCP snooping, Dynamic ARP Inspection, DoH/DoT |
| Rogue DHCP | hand out a malicious gateway and DNS server | DHCP snooping, port security, 802.1X |
| ARP spoofing | claim to be the gateway or a peer, then relay traffic | Dynamic ARP Inspection, static ARP entries for gateways, edge encryption |
| TCP session hijacking | inject into or reset a session using known sequence numbers | TLS everywhere, segmentation, monitoring for unexpected RSTs |
| SYN flood | half-open connections exhaust the server's table | SYN cookies, connection limits, upstream scrubbing |
| UDP amplification / reflection | spoof the victim's address and trigger large replies from open services | close open resolvers, disable legacy UDP services, BCP 38 ingress filtering |
| Cleartext credential capture | sniff Telnet, FTP, LDAP or HTTP and read the password | replace the protocol with its encrypted equivalent; no exceptions |
| SMB and RDP exposure | scan 445/3389 for missing patches and weak credentials, then move laterally | disable SMBv1, block 445 between workstations, RDP behind VPN or a jump host with MFA |
Key takeaways
- The transport choice is already a security posture. TCP gives you sessions worth hijacking and flooding; UDP gives you spoofing worth amplifying.
- Every protocol in a cleartext column is a finding before it is an incident. Fix the protocol, not the symptom.
- DNS, DHCP and ARP are unauthenticated by design, which is why they are the standard man-in-the-middle toolkit and why they belong in your monitoring.
- An open port is a question, not an answer. Version, configuration and exposure decide whether it is a risk.
- Time is a security control. Break NTP and you break Kerberos, certificate validity and the timeline of your own logs.
Check yourself
- Why can an attacker spoof a UDP source address but not easily spoof one in an established TCP session?
- Name the four DHCP messages in order, and explain what a rogue DHCP server achieves.
- A machine on your LAN is sending an unusual volume of DNS TXT queries. What are two plausible explanations?
- Why is a certificate for an SMB connection worth more than a firewall rule blocking port 139?
- You find Telnet open on a network device. Why is that a finding even if the password is strong?