Prometheus is an open-source monitoring and alerting toolkit that collects, stores, and queries time-series metrics data. It has become the de facto standard for monitoring infrastructure and applications, especially in Kubernetes environments.
Core Architecture
- Pull-Based Collection: Prometheus periodically scrapes metrics from HTTP endpoints exposed by applications and exporters (default: every 15 seconds).
- Time-Series Database: Metrics are stored as time-series data — sequences of timestamped values identified by metric name and key-value labels.
- PromQL: A powerful query language for selecting, filtering, aggregating, and computing over metrics data.
- Alert Manager: Evaluates alerting rules against metrics and routes notifications to email, Slack, PagerDuty, etc.
Key Concepts
- Metrics Endpoint: Applications expose a /metrics HTTP endpoint returning metrics in Prometheus format.
- Exporters: Pre-built adapters that expose metrics from third-party systems (node_exporter for OS metrics, nvidia_gpu_exporter for GPU metrics, mysqld_exporter for MySQL).
- Labels: Key-value pairs that add dimensions to metrics — http_requests_total{method="POST", status="200", model="gpt-4"}.
- Recording Rules: Pre-compute expensive queries and store results as new metrics for dashboard performance.
Prometheus for AI/ML Monitoring
- GPU Metrics: Use DCGM Exporter to collect NVIDIA GPU utilization, memory, temperature, and power consumption.
- Inference Metrics: Track request latency, throughput, queue depth, and error rates for model serving endpoints.
- Custom Metrics: Instrument application code with Prometheus client libraries to expose model-specific metrics (token counts, cache hit rates, quality scores).
Common PromQL Queries
- rate(http_requests_total[5m]) — Requests per second over 5 minutes.
- histogram_quantile(0.99, rate(request_duration_seconds_bucket[5m])) — p99 latency.
- avg(gpu_utilization) by (instance) — Average GPU utilization per server.
Ecosystem
- Grafana: Primary visualization tool for Prometheus metrics — dashboards, graphs, and alerts.
- Thanos / Cortex / Mimir: Long-term storage and horizontal scaling for Prometheus.
- Kubernetes: Prometheus is the native monitoring solution for Kubernetes via kube-prometheus-stack.
Prometheus is a foundational monitoring tool — if you're running any production infrastructure (especially Kubernetes), Prometheus is almost certainly part of your stack.