PROMETHEUS

Last reviewed: 2026-06-16

Purpose: Comprehensive knowledge base article covering Prometheus architecture, PromQL fundamentals, all functions and operators, and a library of practical production queries.

Architecture Overview
PromQL Data Types
Selectors & Matchers
Modifiers: Offset & @
Binary Operators
Aggregation Operators
PromQL Functions by Category
Counter Functions
Gauge Functions
Histogram Functions
Aggregation Helpers
Time & Date Functions
Label Manipulation Functions
Math & Utility Functions
Common Queries

Architecture Overview

Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. Its core components:

Component	Role
Prometheus Server	Scrapes and stores time-series data, evaluates rules, serves queries
Targets	Endpoints exposing metrics (typically `/metrics` via HTTP)
Service Discovery	Automatically finds targets (Kubernetes, Consul, DNS, file-based, etc.)
Alertmanager	Handles deduplication, grouping, silencing, and routing of alerts
Pushgateway	Accepts short-lived job metrics for scraping (used for batch/cron jobs)
Exporters	Third-party agents that translate non-Prometheus metrics (node_exporter, blackbox_exporter, etc.)
Recording Rules	Pre-compute expensive queries for faster dashboard rendering
Alerting Rules	Define alert conditions based on PromQL expressions

Data model: Every time series is identified by a metric name and a set of key-value labels. Samples are (timestamp, value) pairs.

Pull model: Prometheus scrapes targets on a schedule (configured via scrape_interval), unlike push-based systems.

PromQL Data Types

PromQL expressions evaluate to one of four data types:

Type	Description	Example
Instant Vector	A set of time series with a single sample per series at the current evaluation timestamp	`node_cpu_seconds_total`
Range Vector	A set of time series with a range of samples over a time window	`node_cpu_seconds_total[5m]`
Scalar	A single numeric value	`42`, `3.14`
String	A string value (rarely used, only in某些 functions)	`"info"`

Instant vector is the default; you use it for comparisons, aggregations, and most queries.

Range vector is obtained by appending a duration in square brackets: [5m], [1h], [30s]. Required by rate()-family functions.

Selectors & Matchers

Metric Name Selector

The simplest selector is just a metric name:

prometheus_http_requests_total

Returns all time series with that metric name.

Label Matchers

Filter series by label values using matchers inside curly braces {}:

Matcher	Meaning	Example
`=`	Equals	`{job="prometheus"}`
`!=`	Not equals	`{job!="prometheus"}`
`=~`	Regex matches	`{job=~"node\|prometheus"}`
`!~`	Regex not matches	`{job!~".*exporter"}`

prometheus_http_requests_total{handler="/api/v1/query", code="200"}

Implicit `name` Matcher

The metric name itself is a label __name__. These are equivalent:

prometheus_http_requests_total
{__name__="prometheus_http_requests_total"}

Modifiers: Offset & @

Offset Modifier

Shift a query backward in time. Useful for comparing current values to a previous period.

# Memory usage now vs. 1 hour ago
node_memory_MemTotal_bytes - (node_memory_MemFree_bytes offset 1h)

# Rate of requests from 5 minutes ago
rate(prometheus_http_requests_total[5m] offset 5m)

@ Modifier

Evaluate a query at a specific Unix timestamp (in seconds).

# The value of this metric at exactly 2025-01-01 00:00:00 UTC
prometheus_http_requests_total @ 1735689600

Can be combined with offset and range vectors.

Binary Operators

Arithmetic Operators

Operator	Meaning
`+`	Addition
`-`	Subtraction
`*`	Multiplication
`/`	Division
`%`	Modulo
`^`	Power / exponentiation

# CPU utilization percentage
(1 - avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[1m]))) * 100

# Memory percentage used
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100

Comparison Operators

Operator	Meaning
`==`	Equal
`!=`	Not equal
`>`	Greater than
`<`	Less than
`>=`	Greater or equal
`<=`	Less or equal

Comparison operators produce 0 (false) or 1 (true):

# Which nodes have less than 10% free space?
node_filesystem_avail_bytes / node_filesystem_size_bytes < 0.1

Logical / Set Operators

Operator	Meaning
`and`	Intersection — returns LHS elements matching RHS
`or`	Union — returns LHS elements, adding RHS elements not in LHS
`unless`	Subtraction — returns LHS elements not present in RHS

# Nodes that are up AND have high CPU
up == 1 and (1 - rate(node_cpu_seconds_total{mode="idle"}[5m])) > 0.9

# All up targets, filling in missing ones from scrape_duration
up or scrape_duration_seconds

Vector Matching

When binary operators involve two vectors, Prometheus matches them by labels:

One-to-one (default): same labels on both sides
Many-to-one / One-to-many: use on or ignoring with group_left/group_right

# Many-to-one: divide by total count, ignoring 'mode'
rate(node_cpu_seconds_total[1m]) / ignoring(mode) group_left sum(rate(node_cpu_seconds_total[1m]))

Aggregation Operators

Aggregation operators combine series across dimensions. They always produce an instant vector.

Basic form: <aggregation>([parameter,] <expression>) [by|without (<label list>)]

Operator	Description
`sum`	Sum over dimensions
`avg`	Average over dimensions
`min`	Minimum value
`max`	Maximum value
`count`	Count of series
`stddev`	Population standard deviation
`stdvar`	Population standard variance
`topk`	K largest values (returns series, not scalar)
`bottomk`	K smallest values
`quantile`	φ-quantile (0 ≤ φ ≤ 1)
`count_values`	Count of series with each unique value

`by` vs `without`

by (<labels>) — group by the specified labels only
without (<labels>) — group by all labels except those listed

# Total requests per job
sum by(job) (prometheus_http_requests_total)

# Average CPU per instance (drops 'mode' label)
avg without(mode) (rate(node_cpu_seconds_total[5m]))

Examples for Each Aggregation Operator

# sum — total HTTP requests across all handlers
sum(prometheus_http_requests_total)

# avg — average request rate per instance
avg by(instance) (rate(prometheus_http_requests_total[5m]))

# min — node with lowest free memory
min by(instance) (node_memory_MemAvailable_bytes)

# max — node with highest free memory
max by(instance) (node_memory_MemAvailable_bytes)

# count — how many distinct handler/code combinations have errors
count by(handler) (prometheus_http_requests_total{code=~"5.."})

# topk — top 5 CPU-consuming processes
topk(5, rate(process_cpu_seconds_total[1m]))

# bottomk — bottom 10 by free memory
bottomk(10, node_memory_MemAvailable_bytes)

# quantile — 95th percentile of request duration (from a Summary/Histogram)
quantile(0.95, rate(http_request_duration_seconds_sum[5m]) / rate(http_request_duration_seconds_count[5m]))

# stddev — standard deviation of request rates across instances
stddev by(job) (rate(prometheus_http_requests_total[5m]))

# stdvar — variance of request rates
stdvar by(job) (rate(prometheus_http_requests_total[5m]))

# count_values — how many instances report each CPU count value
count_values("cpu_count", node_cpu_seconds_total{mode="idle"})

PromQL Functions by Category

Counter Functions

Counters are monotonically increasing. Use these functions to derive rates and deltas.

`rate()`

Signature: rate(v range-vector) -> instant-vector

Description: Calculates the per-second average rate of increase of a counter over the specified time window. Handles counter resets correctly. The most commonly used PromQL function.

# HTTP requests per second over the last 5 minutes
rate(prometheus_http_requests_total[5m])

Explanation: Takes the counter's value at start and end of the 5m window, computes (end - start) / 300 seconds.

`irate()`

Signature: irate(v range-vector) -> instant-vector

Description: Calculates the instantaneous rate based on the last two data points in the range window. More volatile than rate(), useful for spotting spikes.

# Instantaneous HTTP requests per second (last two samples)
irate(prometheus_http_requests_total[1m])

Explanation: Takes only the last two samples in the 1m window. If the scrape interval is 15s, this gives rate over ~15s rather than the full 1m.

`increase()`

Signature: increase(v range-vector) -> instant-vector

Description: Returns the total increase in a counter over the specified time window. Handles resets.

# Total HTTP requests in the last hour
increase(prometheus_http_requests_total[1h])

Explanation: Computes how many requests happened in the last hour by subtracting the counter value at the start from the value at the end. Equivalent to rate(metric[1h]) * 3600.

Gauge Functions

Gauges can go up and down. These functions analyze their behavior.

`delta()`

Signature: delta(v range-vector) -> instant-vector

Description: Computes the difference between the first and last value of a gauge over the range window. Does not handle counter resets.

# Temperature change in the last 15 minutes
delta(sensor_temperature_celsius[15m])

Explanation: Takes the last sample minus the first sample over 15m. If the current temperature is 25°C and 15m ago it was 22°C, result is 3.

`deriv()`

Signature: deriv(v range-vector) -> instant-vector

Description: Calculates the per-second derivative of a gauge using simple linear regression over the range window.

# Rate of memory consumption (bytes per second)
deriv(node_memory_MemAvailable_bytes[10m])

Explanation: Fits a straight line to the samples in the 10m window and returns the slope in bytes/second. Negative means memory is being consumed (available decreasing).

`predict_linear()`

Signature: predict_linear(v range-vector, t scalar) -> instant-vector

Description: Predicts the value of a gauge t seconds from now using linear regression over the range window.

# Predicted disk usage in 6 hours (21600 seconds)
predict_linear(node_filesystem_avail_bytes[1h], 21600)

Explanation: Using the last hour of data, fits a line and extrapolates 6 hours into the future. Useful for forecasting disk space exhaustion.

`idelta()`

Signature: idelta(v range-vector) -> instant-vector

Description: Calculates the difference between the last two samples in a range vector. Useful for finding the most recent change.

# Most recent change in memory available
idelta(node_memory_MemAvailable_bytes[5m])

Explanation: Takes the last sample minus the second-to-last sample in the 5m window. If samples are 15s apart, shows the change over ~15s.

Histogram Functions

`histogram_quantile()`

Signature: histogram_quantile(φ scalar, b instant-vector) -> instant-vector

Description: Calculates the φ-quantile (0 ≤ φ ≤ 1) from a native Histogram or a classic Histogram metric (bucketed). For classic histograms, the second argument must be a rate() of the histogram bucket's _bucket metric.

# 99th percentile request latency (classic histogram)
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

Explanation: Takes the rate of each latency bucket, sums across all dimensions (keeping le), and computes the 99th percentile. le is the "less than or equal" upper bound of each bucket.

# 50th, 90th, and 99th percentiles in one query
histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket[5m])) by (le)),
histogram_quantile(0.90, sum(rate(http_request_duration_seconds_bucket[5m])) by (le)),
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

Aggregation Helpers

`count_values`

Signature: count_values(output-label string, v instant-vector) -> instant-vector

Description: Counts the number of series that have each distinct value of the input metric.

# How many instances report each number of CPUs?
count_values("cpu_count", node_cpu_seconds_total{mode="idle"})

Explanation: If 3 instances have 4 CPUs and 2 instances have 8 CPUs, results are {cpu_count="4"} -> 3 and {cpu_count="8"} -> 2.

Time & Date Functions

`time()`

Signature: time() -> scalar

Description: Returns the current Unix timestamp in seconds (from the Prometheus server's system clock).

# Current server time
time()

# Time elapsed since a metric changed (seconds since last scrape)
time() - process_start_time_seconds

`timestamp()`

Signature: timestamp(v instant-vector) -> instant-vector

Description: Returns the Unix timestamp of each sample in the instant vector.

# When was each target last scraped?
timestamp(up)

Explanation: Each up series has a sample from the last scrape. timestamp() returns the scrape timestamp, so you can identify stale targets.

`changes()`

Signature: changes(v range-vector) -> instant-vector

Description: Returns the number of times the value changed (not counting interval resets) within the range window.

# How many times did the leader election change in the last hour?
changes(etcd_server_leader_changes_seen_total[1h])

`day_of_week()`

Signature: day_of_week(v instant-vector) -> instant-vector

Description: Returns the day of the week (0=Sunday, 1=Monday, ..., 6=Saturday) for each sample's timestamp.

# Is it Monday?
day_of_week(time()) == 1

`days_in_month()`

Signature: days_in_month(v instant-vector) -> instant-vector

Description: Returns the number of days in the month for each sample's timestamp.

# Current month length
days_in_month(time())

`hour()`

Signature: hour(v instant-vector) -> instant-vector

Description: Returns the hour (0–23) of the day for each sample's timestamp.

# Active during business hours only
hour(time()) >= 9 and hour(time()) < 17

`minute()`

Signature: minute(v instant-vector) -> instant-vector

Description: Returns the minute (0–59) for each sample's timestamp.

# Every 30-minute mark
minute(time()) % 30 == 0

`month()`

Signature: month(v instant-vector) -> instant-vector

Description: Returns the month (1–12) for each sample's timestamp.

# End-of-quarter months
month(time()) % 3 == 0

`year()`

Signature: year(v instant-vector) -> instant-vector

Description: Returns the four-digit year for each sample's timestamp.

# Current year
year(time())

Label Manipulation Functions

`label_replace()`

Signature: label_replace(v instant-vector, dst_label string, replacement string, src_label string, regex string) -> instant-vector

Description: Copies and transforms a label value via regex replacement. Creates a new time series set; does not modify the original.

# Extract the instance IP (strip port)
label_replace(up, "ip", "$1", "instance", "([^:]+):.*")

Explanation: For instance="10.0.0.1:9090", the regex captures 10.0.0.1 into $1 and assigns it to a new label ip.

`label_join()`

Signature: label_join(v instant-vector, dst_label string, separator string, src_labels ...string) -> instant-vector

Description: Joins the values of multiple source labels into a single destination label, separated by the given separator.

# Create a combined "job_instance" label
label_join(up, "job_instance", "/", "job", "instance")

Explanation: If job="node" and instance="10.0.0.1:9100", the result gets job_instance="node/10.0.0.1:9100".

Math & Utility Functions

`abs()`

Signature: abs(v instant-vector) -> instant-vector

Description: Returns the absolute value of each sample.

# Absolute deviation from target memory usage
abs(node_memory_MemTotal_bytes - 8589934592)

`absent()`

Signature: absent(v instant-vector) -> instant-vector

Description: Returns an empty series if the expression has results, or a 1 if the expression has no results. Useful for alerting on missing metrics.

# Alert if a specific job's metric disappears
absent(up{job="myapp"})

Explanation: If up{job="myapp"} has no data, returns {job="myapp"} -> 1. If data exists, returns nothing.

`ceil()`

Signature: ceil(v instant-vector) -> instant-vector

Description: Rounds each sample up to the nearest integer.

# Round up CPU percentage
ceil((1 - rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

`floor()`

Signature: floor(v instant-vector) -> instant-vector

Description: Rounds each sample down to the nearest integer.

# Round down to nearest whole GB
floor(node_memory_MemTotal_bytes / 1073741824)

`clamp()`

Signature: clamp(v instant-vector, min scalar, max scalar) -> instant-vector

Description: Clamps each sample's value between a minimum and maximum.

# Clamp CPU percentage to 0-100 range
clamp(100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100), 0, 100)

`clamp_min()`

Signature: clamp_min(v instant-vector, min scalar) -> instant-vector

Description: Clamps values to be at least min. Values below min are raised to min.

`clamp_max()`

Signature: clamp_max(v instant-vector, max scalar) -> instant-vector

Description: Clamps values to be at most max. Values above max are lowered to max.

`exp()`

Signature: exp(v instant-vector) -> instant-vector

Description: Returns the exponential (e^x) of each sample.

# Exponential of the metric value
exp(prometheus_http_requests_total)

`log()`

Signature: log(v instant-vector) -> instant-vector

Description: Returns the natural logarithm of each sample.

`log2()`

Signature: log2(v instant-vector) -> instant-vector

Description: Returns the base-2 logarithm of each sample.

`log10()`

Signature: log10(v instant-vector) -> instant-vector

Description: Returns the base-10 logarithm of each sample.

# Log-scale request rate
log10(rate(prometheus_http_requests_total[5m]) + 1)

`sqrt()`

Signature: sqrt(v instant-vector) -> instant-vector

Description: Returns the square root of each sample.

# Root of counter rate
sqrt(rate(prometheus_http_requests_total[5m]))

`sgn()`

Signature: sgn(v instant-vector) -> instant-vector

Description: Returns the sign of each sample: -1 for negative, 0 for zero, 1 for positive.

# Positive or negative derivative
sgn(deriv(node_memory_MemAvailable_bytes[10m]))

`round()`

Signature: round(v instant-vector, to_nearest=1 scalar) -> instant-vector

Description: Rounds each sample to the nearest integer or to the nearest multiple of to_nearest.

# Round to nearest 0.1
round(rate(prometheus_http_requests_total[5m]), 0.1)

`sort()`

Signature: sort(v instant-vector) -> instant-vector

Description: Returns series sorted ascending by value.

# Instances sorted by error rate (ascending)
sort(rate(prometheus_http_requests_total{code=~"5.."}[5m]))

`sort_desc()`

Signature: sort_desc(v instant-vector) -> instant-vector

Description: Returns series sorted descending by value.

# Instances sorted by error rate (descending)
sort_desc(rate(prometheus_http_requests_total{code=~"5.."}[5m]))

`vector()`

Signature: vector(s scalar) -> instant-vector

Description: Converts a scalar value into an instant vector with no labels. Useful in recording rules or calculations where a scalar needs to be treated as a vector.

# A constant 1 as a vector (often used in alerting)
vector(1)

Common Queries

1. Error Rate (Percentage)

# Percentage of HTTP 5xx responses in the last 5 minutes
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
* 100

2. Request Latency P99

# 99th percentile of request duration (classic histogram)
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job)
)

3. Request Latency P50 / P90 / P99 (All at Once)

# Three quantiles in a single query using label_replace trick
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job))
and
histogram_quantile(0.90, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job))
and
histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job))

4. CPU Usage Per Instance (Percentage)

# CPU utilization per instance, as a percentage
(1 - avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m]))) * 100

5. CPU Usage Per Core

# Per-core CPU utilization
(1 - rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100

6. Memory Usage Percentage

# Memory used as percentage of total
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100

7. Disk Space Percentage (Per Filesystem)

# Disk usage percentage, excluding tmpfs and devtmpfs
(node_filesystem_size_bytes{mountpoint!~"^/dev|^/run|^/sys|^/proc"}
- node_filesystem_avail_bytes{mountpoint!~"^/dev|^/run|^/sys|^/proc"})
/
node_filesystem_size_bytes{mountpoint!~"^/dev|^/run|^/sys|^/proc"}
* 100

8. Up / Down Targets

# Number of up targets per job
count by(job) (up == 1)

# Number of down targets per job
count by(job) (up == 0)

# Which specific targets are down?
up == 0

9. Memory Forecasting (Will we run out?)

# Predict available memory in 2 hours (7200s)
predict_linear(node_memory_MemAvailable_bytes[1h], 7200) < 0

Returns series where available memory is predicted to hit zero within 2 hours.

10. Disk Fill Forecasting

# Predict disk space exhaustion in 24 hours (86400s)
predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[4h], 86400) < 0

11. Compare Current vs. Last Week

# Request rate now vs. same time last week
rate(http_requests_total[5m]) / rate(http_requests_total[5m] offset 1w) - 1

Positive values = increase; negative = decrease compared to last week.

12. Top 10 Services by Request Rate

# Top 10 services by HTTP request rate
topk(10, sum by(service) (rate(http_requests_total[5m])))

13. Network I/O Rate

# Network bytes received per second (node_exporter)
rate(node_network_receive_bytes_total[5m])

# Network bytes transmitted per second
rate(node_network_transmit_bytes_total[5m])

14. Scrape Duration and Samples

# Average scrape duration per job
avg by(job) (rate(prometheus_target_interval_length_seconds[5m]))

# Total samples scraped per job
sum by(job) (prometheus_tsdb_head_samples_appended_total)

15. Service Uptime Over the Last 30 Days

# Percentage of time a target was up over the last 30 days
avg_over_time(up{job="myapp"}[30d]) * 100

16. Rate of Change in Request Errors (Acceleration)

# Second derivative of error count (rate of change of error rate)
deriv(rate(http_requests_total{status=~"5.."}[5m])[10m])

A positive value means errors are accelerating.

17. Alert: Metric Disappeared

# Fire if the metric has no data for 5 minutes
absent(up{job="critical-app"})

18. Application Restart Detection

# Has this process restarted in the last 15 minutes?
changes(process_start_time_seconds[15m]) > 0

19. Memory per Process (with node_exporter)

# Top 10 processes by RSS memory
topk(10, process_resident_memory_bytes)

20. Saturation: CPU Load vs. CPU Count

# Load average divided by number of CPUs (1.0 = fully saturated)
node_load1 / count by(instance)(node_cpu_seconds_total{mode="idle"})

Tips & Best Practices

Practice	Recommendation
Use `rate()` over `irate()` for dashboards	`rate()` smooths spikes; `irate()` is good for troubleshooting spiky behavior
Always use `rate()` with histograms	Pass `rate(..._bucket[5m])` into `histogram_quantile()`, never raw counters
Choose the right range window	For `rate()`, use at least 4× the scrape interval (e.g., `[5m]` for 15s scrape)
Use recording rules	Pre-compute expensive queries that appear on multiple dashboards
Label hygiene	Keep label cardinality low; avoid labels with unlimited values (user IDs, emails, IPs)
Prefer `by` over `without`	`by` makes your intent explicit and avoids surprises if labels change
Always compare gauges with offset	Use `offset` to compare current vs. previous period for seasonality-aware alerts
Alert on `absent()` for critical metrics	Missing metrics are often worse than bad metrics

PROMETHEUS

Table of Contents

Architecture Overview

PromQL Data Types

Selectors & Matchers

Metric Name Selector

Label Matchers

Implicit __name__ Matcher

Modifiers: Offset & @

Offset Modifier

@ Modifier

Binary Operators

Arithmetic Operators

Comparison Operators

Logical / Set Operators

Vector Matching

Aggregation Operators

by vs without

Examples for Each Aggregation Operator

PromQL Functions by Category

Counter Functions

rate()

irate()

increase()

Gauge Functions

delta()

deriv()

predict_linear()

idelta()

Histogram Functions

histogram_quantile()

Aggregation Helpers

count_values

Time & Date Functions

time()

timestamp()

changes()

day_of_week()

days_in_month()

hour()

minute()

month()

year()