Skip to content

ETL

Last reviewed: 2026-05-29

Purpose: Extract, Transform, Load data pipeline training — covering data ingestion pipelines, transformations, and batch/stream processing patterns.

Contents

Overview

This folder covers ETL (Extract, Transform, Load) pipeline design and implementation, with practical references to the streaming data pipelines in the training ecosystem.

ETL Pipeline Concepts

  • Extract — Pulling data from sources (databases, APIs, SSE streams, flat files)
  • Transform — Cleaning, normalizing, enriching, and structuring data (JSON parsing, type casting, field extraction)
  • Load — Writing to targets (OLAP databases, data lakes, message queues)

Real-World Examples

The training ecosystem includes working examples of ETL patterns:

Streaming ETL (Wikimedia → Kafka → Pinot)

  1. Extract — SSE client (wiki.py) consumes live Wikipedia event stream from stream.wikimedia.org
  2. Transform — The wiki_to_kafka.py script serializes events (datetime → ISO format, JSON → encoded bytes)
  3. Load — Produce to Kafka topic wiki-events with batching (flush every 100 events)

Batch ETL (JSON Files → Pinot)

  1. Extract — JSON data files (check-in/checkout events, FMEA events, license usage reports)
  2. Transform — Pinot table configs define transform functions using JSONPATH extraction and JSONFORMAT serialization
  3. Load — Batch ingestion via Pinot OFFLINE tables (fnmea-events-table-OFFLINE.json)

Schema Inference

  • InferSchemaFromjson.sh — Automatically infer Pinot schema from JSON data files
  • InferMetaEventFromjson.sh — Infer meta-event schema for complex nested data