Skip to content

GRAFANA

Last reviewed: 2026-06-16

Purpose: Comprehensive Grafana knowledge base — covering architecture, installation, data source integration, dashboard design, alerting, provisioning, user management, service accounts, and production deployment patterns for metrics, logs, and traces observability.

Table of Contents

Architecture Overview

Grafana is an open-source observability and data visualization platform. It connects to multiple data sources and presents data through customizable dashboards, panels, and alerts.

                          +------------------+
                          |   Web Browser    |
                          +--------+---------+
                                   |
                          (HTTP/HTTPS :3000)
                                   |
                          +--------+---------+
                          |     Grafana      |
                          |    (Server)      |
                          +--+---+---+---+---+
                             |   |   |   |
                   +---------+   |   |   +---------+
                   |             |   |               |
            +------+---+   +----+---+----+   +------+-------+
            | Prometheus|   |    Loki    |   |   Jaeger     |
            | (Metrics) |   |  (Logs)    |   |  (Traces)    |
            +----------+   +-----------+   +--------------+

Key components:

Component Role
Grafana Server Web service, authentication, alerting engine, dashboard rendering
Data Sources External time-series, log, trace, and SQL databases queried in real time
Dashboards Collections of panels organized into rows, shared via folders
Alerting Engine Evaluates alert rules and sends notifications through contact points
Provisioning YAML-driven automated configuration baked into the container
Plugins Extend core with new data sources, panels, and apps

Grafana does not store the underlying observability data — it queries data sources on demand when a dashboard is loaded or an alert rule evaluates.

Installation via Docker

Quick Start (Single Container)

docker run -d \
  --name grafana \
  -p 3000:3000 \
  -e GF_SECURITY_ADMIN_PASSWORD=admin \
  -v grafana-storage:/var/lib/grafana \
  grafana/grafana:latest

Access at http://localhost:3000 (default login: admin / admin).

Docker Compose (Full Observability Stack)

version: "3.8"
services:
  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_INSTALL_PLUGINS=grafana-piechart-panel
    volumes:
      - grafana-data:/var/lib/grafana
      - ./provisioning:/etc/grafana/provisioning
    restart: unless-stopped

  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    restart: unless-stopped

  loki:
    image: grafana/loki:latest
    container_name: loki
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    restart: unless-stopped

  jaeger:
    image: jaegertracing/all-in-one:latest
    container_name: jaeger
    ports:
      - "16686:16686"   # UI
      - "4317:4317"     # OTLP gRPC
      - "4318:4318"     # OTLP HTTP
    environment:
      - COLLECTOR_OTLP_ENABLED=true
    restart: unless-stopped

volumes:
  grafana-data:

Configuration via Environment Variables

Grafana settings are overridable via GF_<SECTION>_<KEY> environment variables:

Variable Effect
GF_SERVER_HTTP_PORT Change listening port
GF_SECURITY_ADMIN_PASSWORD Set admin password
GF_SECURITY_ADMIN_USER Set admin username
GF_INSTALL_PLUGINS Comma-separated list of plugins to pre-install
GF_AUTH_ANONYMOUS_ENABLED Enable anonymous access
GF_AUTH_LDAP_ENABLED Enable LDAP authentication
GF_SMTP_ENABLED Enable email for alerting
GF_SMTP_HOST SMTP server address

Persistence Directories

Path Purpose
/var/lib/grafana SQLite DB, sessions, plugin data, user preferences
/etc/grafana/grafana.ini Main config file
/etc/grafana/provisioning/datasources/ YAML datasource provisioning
/etc/grafana/provisioning/dashboards/ YAML dashboard provisioning
/etc/grafana/provisioning/alerting/ YAML alert rule provisioning
/var/log/grafana Grafana logs

Data Source Configuration

Grafana supports 50+ data sources. The most common in an observability stack are Prometheus (metrics), Loki (logs), and Jaeger/Tempo (traces).

Prometheus

Connection details:

Parameter Value
Type Prometheus
URL http://prometheus:9090 (Docker internal) or http://host.docker.internal:9090
Access Server (default) or Browser
Scrape interval 15s (default)
Timeout 60s

Manual setup (UI):

  1. Connections → Data Sources → Add data source
  2. Select Prometheus
  3. Set URL to your Prometheus server
  4. Click Save & Test

PromQL query example in panel editor:

# Rate of HTTP requests per second
rate(http_requests_total[5m])

# 95th percentile latency
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

# CPU usage by instance
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

Loki

Connection details:

Parameter Value
Type Loki
URL http://loki:3100
Max lines 1000 (default)
Derived fields Add regex to extract trace IDs from log lines

Manual setup (UI):

  1. Connections → Data Sources → Add data source
  2. Select Loki
  3. Set URL to your Loki instance
  4. Click Save & Test

LogQL query examples in panels:

# All logs from a specific job in the last hour
{job="my-app"} |= ``

# Error count per minute
sum by (level) (rate({job="my-app"} |= "error"[5m]))

# Filter by trace ID (with derived field linking to Tempo/Jaeger)
{job="my-app"} |= "trace_id=abc123"

# JSON log parsing
{job="my-app"} | json | status >= 500

Derived fields allow clicking a trace ID in a log line to jump directly to the trace in Jaeger or Tempo.

Jaeger / Tempo

Connection details (Jaeger):

Parameter Value
Type Jaeger
URL http://jaeger:16686
Node graph Enabled for service dependency visualization

Connection details (Tempo):

Parameter Value
Type Tempo
URL http://tempo:3200
Trace to logs Link traces to Loki logs via service.name and trace ID

TraceQL query example (Tempo):

{ span.http.method = "GET" && span.http.status_code >= 500 }
| count()

PostgreSQL

Connection details:

Parameter Value
Type PostgreSQL
Host postgres:5432 or localhost:5432
Database mydb
User grafana_reader
TLS/SSL disable (or require for production)
TimescaleDB Check if using TimescaleDB time-series features

SQL query example in panel:

SELECT
  time_bucket('1 hour', created_at) AS time,
  status,
  count(*) AS count
FROM orders
WHERE created_at BETWEEN $__timeFrom AND $__timeTo
GROUP BY time, status
ORDER BY time;

Grafana macro variables for SQL data sources:

Macro Expands To
$__timeFrom Start of query time range (ISO)
$__timeTo End of query time range (ISO)
$__timeFilter(timestamp) timestamp BETWEEN '...' AND '...'
$__interval Dynamic interval based on time range
$__unixEpochFrom() Unix timestamp start
$__unixEpochTo() Unix timestamp end

Dashboard Creation

Panels and Visualizations

Each panel in a dashboard queries one or more data sources and renders a visualization.

Supported visualization types:

Type Use Case
Time series Continuous metrics over time (default for PromQL)
Stat Single numeric value with optional sparkline
Gauge Single value in a radial gauge with thresholds
Bar chart Categorical or time-based bar comparison
Table Tabular data with optional column styling
Pie chart Distribution breakdown (plugin required pre-v11)
Logs Log stream viewer with highlighting
Trace view Waterfall trace visualization
Node Graph Service dependency topology
Heatmap 2D histogram of values over time
Geomap Geographic point/heatmap data
Canvas Free-form custom layout

Panel configuration essentials:

  • Query (A, B, C...): You can layer multiple queries (A, B, C) in a single panel, then transform or combine them.
  • Legend: Show/hide, change placement, modify value display (min, max, avg, current, total).
  • Tooltip: Single or all-series, with optional sort.
  • Thresholds: Color-coded zones (e.g., green < 70, yellow < 90, red >= 90).
  • Data links: Click a data point to navigate to another dashboard or URL with template variables.

Rows and Layout

Rows organize panels into collapsible groups. Best practices:

  • Use one row per concern (e.g., "Resource Usage", "Application Errors", "Business Metrics").
  • Set row repeat for multi-instance dashboards (repeat per value of a template variable).
  • Enable collapsed row for secondary detail panels to reduce visual noise.

Panel sizing tips:

  • Use wide, short panels for time series (they display more history).
  • Use small stat panels for key DORA metrics (deployment frequency, MTTR, change failure rate).
  • Keep the most critical panels in the top-left area (users read left-to-right, top-to-bottom).

Template Variables

Template variables make dashboards dynamic and reusable.

Defining a variable (Settings → Variables → Add variable):

Field Example
Name instance
Type Query
Data source Prometheus
Query label_values(node_uname, instance)
Multi-value Enabled
Include All Enabled
Default All

Common variable queries (Prometheus):

# All instance labels
label_values(instance)

# All job names
label_values(up, job)

# All values of a label filtered by another variable
label_values(up{job="$job"}, instance)

# Dynamic interval variable (Custom type)
5m,10m,30m,1h,3h,6h,12h,1d

Using variables in queries:

# Reference with $variable or [[variable]]
rate(http_requests_total{instance="$instance", job="$job"}[5m])

# Use in dashboard links
/d/miniD/dashboard-name?var-instance=$instance&var-job=$job

Special built-in variables:

Variable Description
$__from / $__to Dashboard time range (epoch ms)
$__rate_interval Optimal rate interval (Prometheus-specific)
$__interval Auto-calculated time grouping interval
$__interval_ms Same as above in milliseconds

Transformations

Transformations let you manipulate data after querying but before visualization. They apply panel-wide.

Common transformations:

Transformation Purpose
Reduce Convert multiple values to single stat (min, max, mean, sum, last)
Calculate field Math between fields (e.g., #A - #B for difference)
Group by Aggregate by label and compute sum/mean
Merge Combine multiple time series with different labels
Organize fields Reorder, rename, hide fields
Add field from calculation Binary operations, percent change, time between
Convert field type String → number, number → string, time
Filter by value Keep rows where value matches condition
Outer join Join multiple query results on a common field
Rename by regex Batch-rename fields using regex capture groups

Example — Convert rate to percentage:

  1. Query A: rate(http_requests_total{status="5xx"}[5m])
  2. Query B: rate(http_requests_total[5m])
  3. Transformation → Add field from calculation → #A / #B * 100
  4. Alias: "Error %"

PromQL Queries in Panels

PromQL (Prometheus Query Language) is the primary query language for Prometheus panels in Grafana.

Selectors and Matchers

# Exact match
http_requests_total{job="api", instance="app-1:8080"}

# Regex match
http_requests_total{job=~"api|web"}

# Negative match
http_requests_total{job!="internal"}

# Negative regex
http_requests_total{instance!~"canary.*"}

Range Vectors

Range vectors add a duration in brackets for functions like rate() and avg_over_time():

rate(http_requests_total[5m])
increase(http_requests_total[1h])
avg_over_time(cpu_usage[15m])
max_over_time(memory_bytes[30m])

Aggregation Operators

# Sum by label
sum by (instance) (rate(http_requests_total[5m]))

# Average across all instances
avg(rate(http_requests_total[5m]))

# Top 3 by value
topk(3, rate(http_requests_total[5m]))

# Count distinct label values
count by (status) (http_requests_total)

# Quantile across instances
quantile(0.99, rate(http_request_duration_seconds[5m]))

Common Panel Queries

CPU usage (percent):

100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

Memory usage (bytes → GB):

node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes

HTTP error ratio:

(
  sum(rate(http_requests_total{status=~"5.."}[5m]))
  /
  sum(rate(http_requests_total[5m]))
) * 100

Request duration (p50, p95, p99):

# p50
histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
# p95
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
# p99
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

Up/down status (target health):

# Count of UP targets
count by (job) (up == 1)

# Count of DOWN targets
count by (job) (up == 0)

Alerting

Grafana Alerting (v8+) is a unified alerting system that replaces legacy dashboard alerts.

Alert Rules

Alert rules are evaluated against data source queries. They can be:

  • Grafana-managed: Evaluated by Grafana's internal alerting engine.
  • Mimir/Loki-managed: Evaluated at the data source (Mimir or Loki ruler).

Rule anatomy:

# Example alert rule (Grafana-managed)
groups:
  - name: HighErrorRate
    rules:
      - alert: HighErrorRate
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total[5m]))
          ) * 100 > 5
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "HTTP 5xx error rate above 5%"
          description: "Error rate is {{ $value | humanize }}% for the last 5 minutes"

Key fields:

Field Description
expr PromQL expression that evaluates to an alert
for How long the condition must hold before firing
labels Custom labels attached to the alert (used for routing)
annotations Human-readable info in notifications

Alert rule evaluation:

  1. Pending: Condition met but for duration not elapsed yet.
  2. Firing: Condition met for the full for duration.
  3. Resolved: Condition no longer met; optional notification sent.

Contact Points

Contact points define how to notify (email, Slack, PagerDuty, webhook, etc.).

Configurable contact point types:

Type Configuration
Email SMTP server configured in grafana.ini
Slack Webhook URL, channel, optional username/icon
PagerDuty Integration key, severity mapping
Webhook POST to arbitrary URL with customizable payload
Discord Webhook URL
Telegram Bot token, chat ID
OpsGenie API key, region
Microsoft Teams Webhook URL
Prometheus Alertmanager URL to external Alertmanager

Webhook payload template (Go template):

{
  "title": "{{ .Title }}",
  "message": "{{ .Message }}",
  "status": "{{ .Status }}",
  "labels": {{ toJSON .Labels }},
  "values": {{ toJSON .Values }},
  "startedAt": "{{ .StartsAt }}"
}

Notification Policies

Notification policies control who gets notified and when. They use a tree-based routing structure:

  • Default policy: Catch-all for alerts without matching labels.
  • Nested policies: Match on label matchers (e.g., severity=critical), with optional timing overrides.

Example policy tree:

Default policy (all alerts)
  → Send to team-ops@ Slack channel
  ├── Matcher: severity=critical
  │   → Send to pagerduty-critical contact point
  │   → Repeat notification every 5 minutes
  ├── Matcher: team=frontend
  │   → Send to frontend-slack contact point
  └── Matcher: team=backend
      → Send to backend-slack contact point

Silences: Mute alert notifications for a specific time window (maintenance, known issues).

User and Team Management

User Roles

Role Permissions
Admin (Server) Full access to all orgs, config, users, and plugins
Editor Create/edit dashboards, folders, alerts; cannot add data sources
Viewer View dashboards; cannot edit
Grafana Admin (legacy) Full server admin with ability to manage users, orgs, licenses
No Basic Role Custom roles (RBAC in Grafana Enterprise)

Team Management

Teams group users for simplified permission management.

Create a team:

  1. Administration → Users and access → Teams → New team
  2. Name the team (e.g., platform-engineering)
  3. Add members by email or username

Assign folder permissions to a team:

  1. Dashboards → Browse → Select a folder
  2. Folder actions → Permissions
  3. Add permission → team: platform-engineering → View / Edit / Admin

Permission levels per folder/dashboard:

Permission Viewer Editor Admin
View ✓ ✓ ✓
Create - ✓ ✓
Edit - ✓ ✓
Delete - ✓ ✓
Manage permissions - - ✓

Provisioning

Provisioning allows declarative YAML configuration of data sources, dashboards, and alert resources. This is the recommended approach for production deployments — no manual UI steps needed.

datasources.yml

Path: /etc/grafana/provisioning/datasources/datasources.yml

apiVersion: 1

datasources:
  # --- Prometheus (Metrics) ---
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false
    jsonData:
      timeInterval: "15s"
      queryTimeout: "60s"
      httpMethod: "POST"
    secureJsonData:
      # If using basic auth:
      # basicAuthUser: prometheus
      # basicAuthPassword: ${PROMETHEUS_PASSWORD}
      # If using bearer token:
      # httpHeaderValue1: "Bearer ${PROMETHEUS_TOKEN}"

  # --- Loki (Logs) ---
  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    editable: false
    jsonData:
      maxLines: 2000
      derivedFields:
        - name: "trace_id"
          matcherRegex: "trace_id=(\\w+)"
          url: "$${__value.raw}"
          datasourceUid: tempo-uid

  # --- Tempo (Traces) ---
  - name: Tempo
    type: tempo
    access: proxy
    url: http://tempo:3200
    editable: false
    uid: tempo-uid
    jsonData:
      tracesToLogs:
        datasourceUid: loki-uid
        tags:
          - "service.name"
          - "instance"
        mappedTags:
          - key: "service.name"
            value: "service"
        spanStartTimeShift: "1h"
        spanEndTimeShift: "-1h"
        filterByTraceID: false
      serviceMap:
        datasourceUid: prometheus-uid
      search:
        hide: false

  # --- Jaeger (Traces, alternative to Tempo) ---
  - name: Jaeger
    type: jaeger
    access: proxy
    url: http://jaeger:16686
    editable: false
    jsonData:
      nodeGraph:
        enabled: true

  # --- PostgreSQL (Business Metrics) ---
  - name: PostgreSQL
    type: postgres
    access: proxy
    url: postgres:5432
    database: observability
    user: grafana_reader
    secureJsonData:
      password: ${POSTGRES_PASSWORD}
    jsonData:
      sslmode: disable
      postgresVersion: 1500
      timescaledb: false

dashboards.yml

Path: /etc/grafana/provisioning/dashboards/dashboards.yml

apiVersion: 1

providers:
  - name: "Observability Dashboards"
    orgId: 1
    folder: "Observability"
    folderUid: ""
    type: file
    disableDeletion: true
    updateIntervalSeconds: 30
    allowUiUpdates: true
    options:
      path: /etc/grafana/provisioning/dashboards
      foldersFromFilesStructure: false

  - name: "Application Dashboards"
    orgId: 1
    folder: "Applications"
    type: file
    disableDeletion: false
    updateIntervalSeconds: 60
    allowUiUpdates: true
    options:
      path: /etc/grafana/provisioning/dashboards/apps
      foldersFromFilesStructure: true

Key fields explained:

Field Description
name Provider name (used in logs, not shown in UI)
folder UI folder name where dashboards appear
disableDeletion Prevent UI deletion of provisioned dashboards
updateIntervalSeconds How often Grafana checks for new/updated JSON files
allowUiUpdates Allow saving changes made in the UI back to the JSON file
foldersFromFilesStructure Create sub-folders matching the filesystem hierarchy

Complete Provisioning Example

Below is a fully working provisioning setup that can be dropped into a Docker volume mounted at /etc/grafana/provisioning/.

Directory structure:

provisioning/
├── datasources/
│   └── datasources.yml
├── dashboards/
│   ├── dashboards.yml
│   ├── infrastructure-metrics.json
│   ├── application-metrics.json
│   └── logs-dashboard.json
└── alerting/
    └── alert-rules.yml

datasources.yml — see the full example above.

dashboards.yml:

apiVersion: 1

providers:
  - name: "Infrastructure"
    orgId: 1
    folder: "Infrastructure"
    type: file
    disableDeletion: true
    updateIntervalSeconds: 30
    allowUiUpdates: false
    options:
      path: /etc/grafana/provisioning/dashboards

alerting rules YAML (Grafana-managed, v11+):

Path: /etc/grafana/provisioning/alerting/alert-rules.yml

apiVersion: 1

groups:
  - name: InfrastructureAlerts
    interval: 30s
    rules:
      - uid: high_cpu_alert
        title: "High CPU Usage"
        condition: "A"
        data:
          - refId: "A"
            relativeTimeRange:
              from: 300
              to: 0
            datasourceUid: prometheus-uid
            model:
              expr: |
                100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90
              intervalMs: 15000
              maxDataPoints: 100
        noDataState: NoData
        execErrState: Error
        for: 5m
        annotations:
          summary: "CPU > 90% on {{ $labels.instance }}"
        labels:
          severity: critical

Docker Compose volume mount to enable provisioning:

services:
  grafana:
    image: grafana/grafana:latest
    volumes:
      - ./provisioning:/etc/grafana/provisioning

Service Account Setup for API Access

Service accounts allow programmatic access to the Grafana API (automation, CI/CD, Terraform).

Create a Service Account (UI)

  1. Administration → Users and access → Service accounts
  2. Click Add service account
  3. Set:
  4. Display name (e.g., deploy-bot)
  5. Role (Viewer, Editor, or Admin)
  6. Click Create
  7. Copy the generated token (shown once only)

Create a Service Account (API)

# 1. Create service account
SA_RESPONSE=$(curl -s -X POST \
  -H "Authorization: Bearer ${GRAFANA_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"name":"deploy-bot","role":"Editor","isDisabled":false}' \
  "http://localhost:3000/api/serviceaccounts")

SA_ID=$(echo "$SA_RESPONSE" | jq -r '.id')

# 2. Create a token for the service account
TOKEN_RESPONSE=$(curl -s -X POST \
  -H "Authorization: Bearer ${GRAFANA_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"name":"deploy-token-1","role":"Editor"}' \
  "http://localhost:3000/api/serviceaccounts/${SA_ID}/tokens")

TOKEN=$(echo "$TOKEN_RESPONSE" | jq -r '.key')
echo "Service account token: ${TOKEN}"

Using the Token (API Examples)

# List dashboards
curl -s -H "Authorization: Bearer ${SA_TOKEN}" \
  "http://localhost:3000/api/search?type=dash-db"

# Create a dashboard
curl -s -X POST \
  -H "Authorization: Bearer ${SA_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"dashboard":{"title":"New Dashboard","panels":[]},"overwrite":true}' \
  "http://localhost:3000/api/dashboards/db"

# List data sources
curl -s -H "Authorization: Bearer ${SA_TOKEN}" \
  "http://localhost:3000/api/datasources"

# Trigger a dashboard refresh
curl -s -X POST \
  -H "Authorization: Bearer ${SA_TOKEN}" \
  "http://localhost:3000/api/admin/provisioning/datasources/reload"

# Create an API key (legacy, v8+ use service accounts instead)
curl -s -X POST \
  -H "Authorization: Bearer ${SA_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"name":"ci-key","role":"Admin","secondsToLive":86400}' \
  "http://localhost:3000/api/auth/keys"

Folders and Permission Management

Folders organize dashboards and scope permissions. Every dashboard belongs to exactly one folder.

Managing Folders

Via UI:

  1. Dashboards → Browse → New folder (top-right)
  2. Name the folder

Via API:

curl -s -X POST \
  -H "Authorization: Bearer ${SA_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"title":"Infrastructure"}' \
  "http://localhost:3000/api/folders"

Folder hierarchy best practices:

Observability/
├── Infrastructure/
│   ├── Node Exporter
│   └── Docker Containers
├── Applications/
│   ├── API Gateway
│   ├── User Service
│   └── Background Workers
├── Logs/
│   └── Application Logs
├── Traces/
│   └── Service Graph
└── Business/
    ├── Revenue Dashboard
    └── User Growth

Setting Permissions

Permission inheritance:

  1. Org-level: Default role (Viewer/Editor/Admin) applies to all dashboards.
  2. Folder-level: Overrides org default for all dashboards in the folder.
  3. Dashboard-level: Overrides folder and org for a single dashboard.

API — Set folder permissions:

curl -s -X POST \
  -H "Authorization: Bearer ${SA_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "items": [
      {"role": "Viewer", "permission": 1},
      {"teamId": 1, "permission": 2},
      {"userId": 42, "permission": 4}
    ]
  }' \
  "http://localhost:3000/api/folders/${FOLDER_UID}/permissions"

Permission codes:

Code Permission
1 View
2 Edit
4 Admin

Common Dashboard Types

Metrics Dashboards

Purpose: Real-time and historical infrastructure and application metrics.

Typical panels:

Panel Title Query (PromQL) Visualization
CPU Usage 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) Time series
Memory Used (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / 1024^3 Time series (GB)
Disk I/O rate(node_disk_io_time_seconds_total[5m]) Time series
Network Traffic rate(node_network_receive_bytes_total[5m]) / rate(node_network_transmit_bytes_total[5m]) Time series
HTTP Requests/s sum(rate(http_requests_total[5m])) Stat + sparkline
Error Rate % (sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))) * 100 Stat (threshold colored)
p99 Latency histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le)) Time series
Top Endpoints topk(10, sum by (endpoint) (rate(http_requests_total[5m]))) Bar chart
Service Health up Stat (1 = green, 0 = red)

Logs Dashboards

Purpose: Real-time log exploration, error tracking, and log volume monitoring.

Typical panels:

Panel Title Query (LogQL) Visualization
Live Log Stream {job="my-app"} Logs panel
Error Rate sum by (level) (rate({job="my-app"} \|= "error"[5m])) Time series
Log Volume by Level sum by (level) (count_over_time({job="my-app"}[5m])) Bar chart
Top Error Messages topk(10, {job="my-app"} \|= "error" \| json \| sum by (message) (rate[5m])) Table
Logs by Host sum by (instance) (count_over_time({job="my-app"}[5m])) Pie chart / Bar chart
Slow Requests (from logs) {job="my-app"} \| json \| duration > 2 Logs panel (filtered)

Traces Dashboards

Purpose: Distributed tracing — request flow, span durations, service dependencies, error breakdown.

Typical panels:

Panel Title Data Source Visualization
Trace Search Jaeger / Tempo Search UI (service, operation, tags, time range)
Trace Waterfall Jaeger / Tempo Trace view (single trace)
Service Graph Jaeger / Tempo Node graph (service dependency topology)
Span Duration Heatmap Tempo Heatmap (span duration vs. time)
Error Spans by Service Tempo (TraceQL) { status = error } \| count_by(service.name)
Rate by Operation Tempo (TraceQL) { } \| rate() by operation
Trace ID from Logs Loki → derived field → Jaeger/Tempo Logs panel with clickable trace links

TraceQL example queries (Tempo):

# All spans with duration > 500ms
{ duration > 500ms }

# Error spans for a specific service
{ resource.service.name = "checkout-service" && status = error }

# Count of spans by HTTP method
{ span.http.method != "" } | count_by(span.http.method)

# Slowest operations
{ duration > 1s } | rate() by(span.http.route)

Troubleshooting

No Data in Panels

  1. Verify data source connection: Go to Connections → Data Sources → click the data source → Save & Test.
  2. Check time range: Grafana defaults to last 6 hours. Extend to a range where data definitely exists.
  3. Confirm metric names: Run the query directly in Prometheus (http://prometheus:9090/graph) to verify the metric exists.
  4. Label matcher issues: In Grafana, template variables may evaluate to empty strings. Check variable definitions.

Dashboard Not Loading

  1. Check browser console (F12) for JavaScript errors.
  2. Clear browser cache and reload.
  3. Check Grafana logs: docker logs grafana | grep error
  4. Large dashboard fallback: If a dashboard has too many panels, Grafana may time out. Reduce time range or simplify queries.

Provisioning Not Working

  1. File paths: Ensure YAML files are mounted at the correct paths:
  2. /etc/grafana/provisioning/datasources/datasources.yml
  3. /etc/grafana/provisioning/dashboards/dashboards.yml
  4. YAML syntax: Validate with docker run --rm -v $(pwd)/provisioning:/provisioning mikefarah/yq yq eval /provisioning/datasources/datasources.yml
  5. Restart Grafana after adding provisioned files: docker restart grafana
  6. Check Grafana logs on startup for provisioning errors.

Alerting Issues

  1. Check alert rule evaluation state under Alerting → Alert rules.
  2. Verify data source UID is correct in alert rule definitions.
  3. Test contact points using the Test button in Alerting → Contact points.
  4. For email: Ensure SMTP is configured in grafana.ini or via GF_SMTP_* environment variables.

Port Conflicts

# Check what is using port 3000
sudo lsof -i :3000
# Or
ss -tlnp | grep 3000

Reset Admin Password

# Reset via Grafana CLI inside the container
docker exec -it grafana grafana-cli admin reset-admin-password newpassword

References