Skip to content

KAFKA

Last reviewed: 2026-05-29

Purpose: Apache Kafka event streaming platform training — covering core concepts, CLI operations, Docker Compose setup, and event-driven architecture patterns.

Contents

Overview

This folder covers Apache Kafka fundamentals — topics, partitions, brokers, producers, consumers, consumer groups — with practical Docker Compose setup and CLI reference. Includes real usage data from a license management system.

Core Concepts

Topics & Partitions

  • A topic is a stream of data (analogous to a DB table)
  • Topics are divided into partitions
  • Partitions maintain an ordered, immutable sequence of records
  • Records get offset IDs within a partition
  • Data is sent to partitions (round-robin if no key, deterministic via key)

Brokers & Clusters

  • A Kafka cluster consists of multiple brokers (servers)
  • Each broker has a unique ID
  • 3 brokers is a good starting cluster size
  • Topics are spread across brokers for scalability

Replication & HA

  • Topics have a replication factor for high availability
  • Recommended: 2 or 3 replicas
  • Each partition has a leader and ISR (In-Sync Replicas)
  • ZooKeeper determines the leader

Producers

  • Write data to topics
  • Know which broker/partition to write to
  • acks=0 — Fire and forget, no acknowledgment
  • acks=1 — Leader acknowledges
  • acks=all — All replicas acknowledge (safest)
  • Message keys route to specific partitions deterministically

Consumers & Consumer Groups

  • Read data in order from a topic
  • Identify themselves by consumer group name
  • Consumer groups share the load — each partition is read by one consumer in the group
  • Offset tracking is managed per consumer group
  • Offsets are not reset unless explicitly configured

Docker Compose Setup

kafka-compose.yaml (Confluent Images)

services:
  zookeeper: confluentinc/cp-zookeeper:latest (port 22181)
  kafka: confluentinc/cp-kafka:latest (port 29092)
    - KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://192.168.122.218:29092
    - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

Single-node setup with ZooKeeper and one Kafka broker. Uses dual listeners — internal (Docker network) and external (host network).

Kafka CLI Reference

Topic Management

# Create topic
kafka-topics.bat --zookeeper 127.0.0.1:2181 --topic first_topic --create --partitions 3 --replication-factor 1

# List topics
kafka-topics.bat --zookeeper 127.0.0.1:2181 --list

# Describe topic
kafka-topics.bat --zookeeper 127.0.0.1:2181 --topic first_topic --describe

Produce & Consume

# Produce with keys
kafka-console-producer --broker-list 127.0.0.1:9092 --topic first_topic \
  --property parse.key=true --property key.separator=,

# Consume with keys (from beginning)
kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic first_topic \
  --from-beginning --property print.key=true --property key.separator=,

Consumer Groups

# List consumer groups
kafka-consumer-groups --bootstrap-server 127.0.0.1:9092 --list

# Describe group
kafka-consumer-groups --bootstrap-server 127.0.0.1:9092 --group my-group --describe

# Reset offsets (to earliest)
kafka-consumer-groups --bootstrap-server 127.0.0.1:9092 \
  --reset-offsets --to-earliest --execute --topic first_topic

Event-Driven Design

Reference: 20220311-EB-Designing_Event_Driven_Systems.pdf (5.7 MB)

PDF book on event-driven architecture patterns, covering event sourcing, CQRS, and event-driven microservices.

Real Usage Data: reporting.json (577 KB)

Large JSON file containing license check-in/check-out events with fields: - feature_name, host_name, feature_version - user_name, user_hostname, user_ipaddr - lic_count (count of licenses), duration_secs (session length) - event_time, linked_event_time (epoch timestamps) - daemon (license vendor daemon, e.g., ptc_d)

Reference: kafka.txt

Concise notes covering Kafka concepts, CLI commands, and configuration tips.

Key Skills Covered

  • Topic creation, partitioning, and replication
  • Producer ack modes (0, 1, all)
  • Consumer group offset management
  • Keys for deterministic partition routing
  • Docker Compose setup for dev Kafka clusters
  • Event-driven architecture principles