Skip to content

PINOT

Last reviewed: 2026-05-29

Purpose: Apache Pinot real-time OLAP database training — a complete streaming analytics pipeline using Apache Pinot, Kafka, Streamlit, and Plotly Dash for real-time Wikipedia edit monitoring.

Contents

Overview

This folder contains a full production-like real-time analytics pipeline with 29 files of real Python, YAML, and JSON code. The pipeline streams live Wikipedia edits from the Wikimedia EventStreams API, feeds them through Apache Kafka into Apache Pinot, and visualizes them in both Streamlit and Plotly Dash dashboards.

Architecture

Wikimedia EventStreams (SSE)
        │
        ▼
wiki.py (SSE client) → prints raw events
wiki_to_kafka.py → produces events to Kafka topic "wiki-events"
        │
        ▼
  Apache Kafka (wurstmeister/kafka)
        │
        ▼
  Apache Pinot (REALTIME table, Kafka consumer)
        │
        ▼
  ┌─────────────────┬──────────────────┐
  │                 │                  │
  ▼                 ▼                  ▼
Streamlit        Plotly Dash       Pinot SQL
(app.py)        (dashboard_*.py)   (direct queries)

Docker Compose Setup

Two Docker Compose variants orchestrate the full stack:

docker-compose.yml (x86/AMD64)

  • ZooKeeper — zookeeper:3.5.6, port 2181
  • Kafka — wurstmeister/kafka:latest, port 9092 (outside), 9093 (internal)
  • Pinot Controller — apachepinot/pinot:0.12.0, port 9000
  • Pinot Broker — apachepinot/pinot:0.12.0, port 8099
  • Pinot Server — apachepinot/pinot:0.12.0, port 8098

docker-compose-m1.yml (Apple Silicon/ARM64)

  • Identical architecture using apachepinot/pinot:0.12.0-arm64 images
  • Pinot Server also exposes port 8097

Both use Kafka advertised listeners with PLAINTEXT://kafka-wiki:9093 for internal and OUTSIDE://localhost:9092 for external access.

Schema & Table Configs

Wikipedia Schema (schema.json)

The wikipedia schema defines 11 dimension columns and 1 timestamp column:

Column Type Source
id STRING Extracted from meta.id
wiki STRING Direct
user STRING Direct
title STRING Direct
comment STRING Direct
stream STRING Extracted from meta.stream
domain STRING Extracted from meta.domain
topic STRING Extracted from meta.topic
type STRING Direct
uri STRING Extracted from meta.uri
bot BOOLEAN Direct
ts TIMESTAMP timestamp * 1000 (epoch millis)

REALTIME Table (table.json)

  • Kafka topic: wiki-events
  • Broker: kafka-wiki:9093
  • Consumer type: low-level
  • Flush threshold: 1000 rows or 24h or 100MB
  • Transform functions extract nested JSON paths from meta column

Local Table (table-local.json)

  • REALTIME table for local dev
  • Kafka topic: wiki_events
  • Broker: localhost:9092
  • Includes JSONFORMAT(meta) → JSONPATH(...) transform pattern

Usage Event Schema (fnmea-events-table-schema.json)

A separate schema usage_event for license usage tracking: - Dimensions: feature_name, host_name, feature_version, user_name, user_hostname, user_ipaddr, daemon - Metrics: lic_count (FLOAT), duration_secs (FLOAT) - Time: event_time, linked_event_time (LONG, epoch millis)

Usage Event Tables

  • fnmea-events-table.json — REALTIME table consuming from Kafka topic transcript-topic at 10.0.0.17:29092
  • fnmea-events-table-OFFLINE.json — OFFLINE (batch) variant of the usage_event table

Streamlit Dashboard

Three versions of the Streamlit app query Pinot in real-time:

Multi-page Streamlit dashboard with: - Overview — Metric cards showing Changes/Users/Domains per minute with delta indicators, plus a Plotly line chart of trends over the last hour - Who's making changes? — Pie chart of bots vs non-bots, bar charts of top users, top bots, and top non-bots - Where changes were made? — Bar charts grouped by domain and change type - Drill Down By User — Select any user from the top 30 to see their domain and type breakdown - Auto-refresh — Configurable refresh rate (1-60s), timestamp displays

app_v1.py (Minimal, 3KB)

  • Single-page overview with metric cards and time series chart
  • Shows "No data loaded yet" prompt when no events exist

app_v2.py (Auto-refresh, 3KB)

  • Single-page overview with auto-refresh toggle
  • Configurable refresh interval
  • Timestamp display

Python Dependencies (requirements.txt)

sseclient-py, confluent-kafka, urllib3, streamlit, plotly, 
pinotdb, pandas, requests, dash, watchdog

Plotly Dash Dashboard

dashboard_v1.py (3KB)

Dash app connecting to Pinot broker on port 8099. Displays: - Indicator gauges for Changes/Users/Domains with delta - Time series line chart (last 8 minutes)

dashboard_v2.py (4KB)

Enhanced Dash app with live auto-refresh callbacks: - indicators callback — Updates every second via dcc.Interval - time_series callback — Queries last hour of data - latest-timestamp callback — Displays last update time

Uses dash_utils.py for shared indicator/datatable rendering helpers.

dash_utils.py (1KB)

Utility module providing: - as_datatable() — Convert DataFrame to Dash DataTable with styled cells - add_delta_trace() — Add indicator gauge with delta - add_trace() — Add simple number indicator - as_data_table_or_message() — Conditional datatable or text

Utility Scripts

wiki.py (SSE Client)

Consumes live Wikipedia recent changes from https://stream.wikimedia.org/v2/stream/recentchange using SSE (Server-Sent Events) and pretty-prints them. Useful for inspecting raw event data.

wiki_to_kafka.py (Kafka Producer)

Consumes the same SSE stream and publishes each event to Kafka topic wiki-events: - Uses confluent_kafka.Producer with ack callback - Serializes datetime objects to ISO format - Flushes every 100 events for batching - Key: meta.id from each event

Usage Example

# 1. Start the stack
cd pinot-wiki
docker-compose up -d

# 2. Create Pinot table
docker exec pinot-controller-wiki bin/pinot-admin.sh AddTable \
  -tableConfigFile /config/table.json \
  -schemaFile /config/schema.json

# 3. Run Kafka producer
pip install -r requirements.txt
python wiki_to_kafka.py

# 4. Launch Streamlit dashboard
streamlit run app.py

# 5. Or launch Dash dashboard
python dashboard_v1.py

Key Skills Covered

  • Real-time OLAP with Apache Pinot (REALTIME tables, Kafka integration)
  • Schema design for streaming data (dimensions, metrics, timestamps)
  • Kafka producer/consumer patterns (acks, batching, serialization)
  • Docker Compose orchestration for multi-service data pipelines
  • Streamlit dashboard development with PinotDB connector
  • Plotly Dash with live-updating callbacks
  • SSE (Server-Sent Events) consumption for live data streams
  • JSON path extraction transforms for nested event data