Skip to content

OBSERVABILITY

Last reviewed: 2026-05-29

Purpose: Full observability stack configurations and guides — Prometheus, Grafana, Jaeger, OpenTelemetry, Docker Compose, and license management.

Contents

Overview

This folder contains a complete production-grade observability stack for the RL Parser application, with Prometheus, Grafana, Jaeger, and OpenTelemetry Collector orchestrated via Docker Compose. Includes detailed setup guides, provisioning configs (auto-datasources, auto-dashboards), and FlexLM license expiration analysis.

Stack Architecture

RL Parser App
    ↓ (OTLP/gRPC :4317)
OTel Collector (receives, processes, routes)
    ↓           ↓
Prometheus   Jaeger (Elasticsearch backend)
    ↓           ↓
   Grafana (Dashboards + Alerting)

Components: - OpenTelemetry Collector — Receives telemetry via OTLP gRPC/HTTP, processes (batching, memory limiting, resource tagging), exports to Prometheus (metrics) and Jaeger (traces) - Prometheus — Metrics storage, 30-day retention, 10GB size limit, lifecycle API enabled - Jaeger — Distributed tracing with Elasticsearch storage backend, 10-day index retention - Grafana — Dashboard visualization, auto-provisioned datasources and dashboards, admin/admin login - Alertmanager — Prometheus alert routing and notifications

Configuration Files

OpenTelemetry Collector (otel-collector-config.yaml)

  • Receivers: OTLP gRPC (port 4317), OTLP HTTP (port 4318)
  • Processors: Batch (10s timeout, 1024 batch size), Resource (adds service.namespace: flexera), Memory limiter (512MB)
  • Exporters: Prometheus (port 8889, rlparser namespace), Jaeger OTLP, Console logging
  • Pipelines: Metrics pipeline and Traces pipeline with same processor chain

Prometheus (prometheus.yml)

  • Global: 15s scrape/evaluation interval, rlparser-dev cluster label
  • Jobs: otel-collector (port 8889) and prometheus self-scrape
  • External labels: cluster: rlparser-dev, environment: development

Grafana Datasources (datasources.yml)

Auto-provisioned: - Prometheus — Default datasource, URL http://prometheus:9090, 15s time interval - Jaeger — URL http://jaeger:16686

Grafana Dashboards (dashboards.yml)

Auto-provisioning provider: - Folder: RL Parser - Scans /etc/grafana/provisioning/dashboards every 30s - UI updates allowed, deletion disabled

Docker Compose

Main Stack (docker-compose.yml)

Service Image Key Ports Notes
otel-collector otel/opentelemetry-collector-contrib:latest 4317, 4318, 8889 OTLP receiver + Prometheus exporter
elasticsearch docker.elastic.co/elasticsearch:8.11.0 9200 Jaeger trace storage, 512MB heap
prometheus prom/prometheus:latest 7070 (maps to 9090) 30d retention, 10GB limit
jaeger jaegertracing/all-in-one:latest 16686 Elasticsearch backend, 10d cleanup
grafana grafana/grafana:latest 6000 (maps to 3000) Auto-provisioned, admin/admin
alertmanager prom/alertmanager:latest 9093 Alert routing

Variants (dated 06MAY2026)

  • docker-compose_V1.1.yml (4KB) — Earlier version of the full stack
  • old_docker-compose (Copy).yml (2.6KB) — Archived earlier variant

All services share a custom bridge network (observability).

Setup Guides

GRAFANA_SETUP_GUIDE.md (35 KB)

Complete guide to setting up Grafana with Docker for RL Parser observability. Covers: - What is Grafana and why use it - Architecture overview with diagrams - Prerequisites (Docker Desktop, WSL2) - Quick start with Docker Compose - Data source configuration (Prometheus, Jaeger) - Creating dashboards (drag-and-drop, queries, panels) - Pre-built RL Parser dashboard (Pub/Sub publish rates, latency percentiles p50/p95/p99, error rates, event distribution) - Querying metrics with PromQL - Viewing distributed traces - Alerting setup (notification channels, thresholds) - Troubleshooting (no data, port conflicts) - Production considerations (auth, data retention, high availability)

OPENTELEMETRY_SETUP_GUIDE.md (29 KB)

Complete OpenTelemetry integration guide for the RL Parser application. Covers: - What is OpenTelemetry (vendor-neutral observability framework) - Core concepts: Traces, Spans, Metrics (Counter, Gauge, Histogram), Logs - OpenTelemetry components: SDK, Exporter, Collector, Instrumentation - Architecture (OTel SDK → Collector → Prometheus/Jaeger) - Installation and dependencies (Go OTel packages) - Configuration (OTLP endpoint, service name, sample rate) - Application integration code examples (metrics counters/histograms, tracing spans) - Viewing telemetry in Grafana and Jaeger - Troubleshooting (connection refused, no traces, high memory) - Best practices (naming conventions, span attributes, sampling, error handling)

License Management

license_expiry.md (6 KB)

Technical deep-dive into FlexLM license expiration behavior. Covers: - Feature expiration: entire feature becomes unavailable at midnight local time - Increment expiration: capacity reduction (e.g., 10 → 6 licenses) - Real log output from lmstat -a before, during, and after expiration - Detailed license server daemon log showing: OUT/IN cycling, DENIED errors (-10,32), auto-reread, EXPIRED status, FEATURE_EXPIRED unsupported status - Report log (.rl) file analysis

Grafana-Prometheus-Info.pdf (2 MB)

Overview PDF on Grafana and Prometheus integration fundamentals.