prometheus

**Prometheus** is an open-source **monitoring and alerting toolkit** that collects, stores, and queries time-series metrics data. It has become the **de facto standard** for monitoring infrastructure and applications, especially in Kubernetes environments. **Core Architecture** - **Pull-Based Collection**: Prometheus periodically **scrapes** metrics from HTTP endpoints exposed by applications and exporters (default: every 15 seconds). - **Time-Series Database**: Metrics are stored as time-series data — sequences of timestamped values identified by metric name and key-value labels. - **PromQL**: A powerful query language for selecting, filtering, aggregating, and computing over metrics data. - **Alert Manager**: Evaluates alerting rules against metrics and routes notifications to email, Slack, PagerDuty, etc. **Key Concepts** - **Metrics Endpoint**: Applications expose a `/metrics` HTTP endpoint returning metrics in Prometheus format. - **Exporters**: Pre-built adapters that expose metrics from third-party systems (node_exporter for OS metrics, nvidia_gpu_exporter for GPU metrics, mysqld_exporter for MySQL). - **Labels**: Key-value pairs that add dimensions to metrics — `http_requests_total{method="POST", status="200", model="gpt-4"}`. - **Recording Rules**: Pre-compute expensive queries and store results as new metrics for dashboard performance. **Prometheus for AI/ML Monitoring** - **GPU Metrics**: Use **DCGM Exporter** to collect NVIDIA GPU utilization, memory, temperature, and power consumption. - **Inference Metrics**: Track request latency, throughput, queue depth, and error rates for model serving endpoints. - **Custom Metrics**: Instrument application code with Prometheus client libraries to expose model-specific metrics (token counts, cache hit rates, quality scores). **Common PromQL Queries** - `rate(http_requests_total[5m])` — Requests per second over 5 minutes. - `histogram_quantile(0.99, rate(request_duration_seconds_bucket[5m]))` — p99 latency. - `avg(gpu_utilization) by (instance)` — Average GPU utilization per server. **Ecosystem** - **Grafana**: Primary visualization tool for Prometheus metrics — dashboards, graphs, and alerts. - **Thanos / Cortex / Mimir**: Long-term storage and horizontal scaling for Prometheus. - **Kubernetes**: Prometheus is the native monitoring solution for Kubernetes via **kube-prometheus-stack**. Prometheus is a **foundational monitoring tool** — if you're running any production infrastructure (especially Kubernetes), Prometheus is almost certainly part of your stack.