KAFKA

Last reviewed: 2026-09-17

Purpose: Apache Kafka event streaming platform — core concepts, topic design, producer and consumer semantics, Docker Compose setup, CLI reference, and event-driven architecture patterns.

---

Table of Contents

• Overview
• Core Concepts
• Topic Design
• Docker Compose Setup
• Kafka CLI Reference
• Producer & Consumer Patterns
• Event-Driven Design
• Topics Covered

---

Overview

Apache Kafka is a distributed, partitioned, replicated commit log used as the backbone of event-driven systems: it ingests events from many producers, retains them durably for a configurable period, and lets any number of consumer groups replay them independently. Typical uses are stream processing, log and metric aggregation, change-data-capture pipelines, and decoupling services through an event log rather than point-to-point requests.

---

Core Concepts

Topics & Partitions

• A topic is a named stream of records — the closest analogue to a database table
• Topics are divided into partitions, the unit of parallelism and ordering
• Each partition is an ordered, immutable append-only log; records get sequential offsets
• A record's key determines its partition (hash of key), so all events for one key stay ordered
• With no key, records are distributed round-robin (sticky partitioning in modern clients)

Brokers & Clusters

• A broker is one Kafka server; a cluster is a set of brokers with unique IDs
• Three brokers is a sensible starting point for production; a single broker is fine for development
• Partitions of a topic are spread across brokers, so throughput scales with broker count

Replication & High Availability

• Each topic has a replication factor — 3 is standard in production, 2 for smaller clusters
• Every partition has one leader (serving reads and writes) and followers forming the ISR
• min.insync.replicas plus acks=all is what actually guarantees no data loss on failover

Producers

• Write records to topic partitions; the client handles batching, compression and retries
• acks=0 — fire and forget, no durability guarantee
• acks=1 — leader acknowledges; lost if the leader dies before replication
• acks=all — all in-sync replicas acknowledge; the safe default for important data
• enable.idempotence=true prevents duplicate records from producer retries
• Keys route deterministically to a partition; use them whenever ordering matters

Consumers & Consumer Groups

• Consumers read sequentially from a partition and commit offsets to track progress
• Members of one consumer group share partitions — each partition is read by exactly one member
• Group size is capped in useful terms by partition count: extra consumers sit idle
• Offsets are retained per group; a new group starts from earliest, latest, or a timestamp
• Rebalancing redistributes partitions when members join or leave; cooperative rebalancing minimises pauses

---

Topic Design

Decision	Guidance
Partition count	Set by target throughput (partitions per broker) — it can be increased but not decreased
Replication factor	3 in production; never 1 for data you cannot lose
Retention	retention.ms / retention.bytes; use compacted topics for keyed state
Cleanup policy	delete (time/size based) or compact (keep the latest value per key)
Naming	<domain>.<entity>.<event> (e.g. orders.payment.completed) keeps topics discoverable
Message size	Default cap is 1 MB; large payloads belong in object storage with a reference in the event

---

Docker Compose Setup

KRaft mode removes the ZooKeeper dependency and is the default in current Kafka releases. For older images, run ZooKeeper alongside a single broker.

# compose.yaml — single-node Kafka for development (KRaft mode)
services:
  kafka:
    image: apache/kafka:3.7.0
    ports:
      - "29092:29092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: INTERNAL://:9092,CONTROLLER://:9093,EXTERNAL://:29092
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,EXTERNAL://192.168.1.50:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
    volumes:
      - kafka-data:/var/lib/kafka/data

volumes:
  kafka-data:

Dual listeners matter: containers on the Compose network use kafka:9092, while clients running on the host use the advertised EXTERNAL address on port 29092. Publishing the wrong advertised listener is the usual cause of "connection to node -1 could not be established".

docker compose up -d
docker compose logs -f kafka
docker compose exec kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka:9092 --list

---

Kafka CLI Reference

# Topic management
kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --create \
  --topic orders.payment.completed --partitions 3 --replication-factor 1
kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --list
kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --topic orders.payment.completed --describe
kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --alter \
  --topic orders.payment.completed --partitions 6
kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --delete --topic scratch.events

# Produce
kafka-console-producer.sh --bootstrap-server 127.0.0.1:9092 --topic first_topic \
  --property parse.key=true --property key.separator=,

# Consume (with keys, from the beginning)
kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic first_topic \
  --from-beginning --property print.key=true --property key.separator=,

# Consumer groups
kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --list
kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --group my-group --describe  # LAG column
kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --group my-group \
  --reset-offsets --to-earliest --execute --topic first_topic

LAG in the group description is the key operational metric: a steadily growing lag means consumers cannot keep up with production.

---

Producer & Consumer Patterns

from confluent_kafka import Producer

producer = Producer({
    "bootstrap.servers": "127.0.0.1:9092",
    "enable.idempotence": True,
    "acks": "all",
    "linger.ms": 20,
})

def delivered(err, msg):
    if err:
        print(f"delivery failed: {err}")

for event in events:
    producer.produce("orders.payment.completed", key=event["order_id"],
                     value=json.dumps(event).encode(), callback=delivered)
producer.flush()     # block until the queue is drained

from confluent_kafka import Consumer, KafkaError

consumer = Consumer({
    "bootstrap.servers": "127.0.0.1:9092",
    "group.id": "fraud-detection",
    "auto.offset.reset": "earliest",
    "enable.auto.commit": False,       # commit only after processing succeeds
})
consumer.subscribe(["orders.payment.completed"])

while True:
    msg = consumer.poll(1.0)
    if msg is None or msg.error():
        continue
    process(msg.value())
    consumer.commit(asynchronous=True)

Commit offsets after the side effect succeeds, not before — that is the difference between at-least-once processing and silent data loss.

---

Event-Driven Design

Delivery semantics

• At-most-once — commit before processing; messages can be lost
• At-least-once — commit after processing; duplicates possible, so consumers must be idempotent
• Exactly-once — transactional producers plus read-process-write transactions, at a throughput cost

Patterns

Pattern	Use
Event notification	Thin events carrying an ID; consumers fetch the current state themselves
Event-carried state transfer	Events carry the full payload so consumers need no callback
Event sourcing	The log is the source of truth; state is derived by replaying events
CQRS	Write model appends events, read model is a projection built from the stream
Outbox	Commit to a database table in the same transaction, then publish from that table
Dead-letter topic	Park unprocessable records for inspection instead of blocking the stream

Operational essentials

• Schema management (Avro/Protobuf with a registry) prevents producers and consumers drifting apart
• Monitor consumer lag, under-replicated partitions, and disk usage per broker
• Keep one topic per event type or aggregate; do not overload a single topic with mixed schemas

---

Topics Covered

• Topic creation, partitioning, replication and retention
• Producer acknowledgement modes (0, 1, all) and idempotent production
• Consumer groups, offset management and lag monitoring
• Keys for deterministic partition routing and ordering guarantees
• Docker Compose deployment with dual internal/external listeners
• Event-driven architecture: event sourcing, CQRS, outbox and delivery semantics