grafana
**Grafana** is the **open-source observability platform that connects to multiple data sources and renders unified dashboards for metrics, logs, and traces** — serving as the "single pane of glass" that teams use to visualize AI infrastructure health, model performance, GPU utilization, and LLM cost analytics without storing data itself.
**What Is Grafana?**
- **Definition**: A multi-source visualization platform that queries data from Prometheus, InfluxDB, Elasticsearch, Loki, Jaeger, PostgreSQL, and dozens of other backends — rendering interactive dashboards with graphs, heatmaps, tables, and alerts.
- **Architecture**: Grafana is a pure visualization layer — it does not store metrics or logs. It queries existing data stores and renders results, making it composable with any monitoring stack.
- **Created By**: Torkel Odegaard (2014), originally forked from Kibana. Now maintained by Grafana Labs with a massive open-source community.
- **Scale**: Used by Netflix, Uber, PayPal, and virtually every major tech company — pre-built dashboards available for every popular AI framework and GPU monitoring stack.
**Why Grafana Matters for AI Teams**
- **Training Run Monitoring**: Visualize loss curves, gradient norms, learning rate schedules, and GPU utilization side-by-side in real time during model training.
- **Inference Dashboard**: Track TTFT (Time to First Token), tokens per second, queue depth, error rates, and cost per query with automatic alerting.
- **GPU Fleet Management**: Monitor temperature, memory usage, power draw, and SM utilization across hundreds of GPUs simultaneously — spot thermal throttling and underutilization instantly.
- **Multi-Source Correlation**: Overlay application metrics (Prometheus), logs (Loki), and traces (Tempo/Jaeger) on the same timeline — find root causes by correlating a latency spike with a log error and a specific trace.
- **Cost Analytics**: Track OpenAI API costs, RunPod GPU hours, and inference infrastructure costs — visualize cost per user, per model, per feature.
**Core Concepts**
**Data Sources**: Grafana's connectivity layer. Configure once, query anywhere:
- Prometheus (metrics time-series)
- Loki (logs — Prometheus-like, but for log streams)
- Tempo (distributed traces)
- InfluxDB (time-series)
- PostgreSQL / MySQL (structured data — query your experiment tracking DB)
- CloudWatch, Azure Monitor, Google Cloud Monitoring
- Elasticsearch / OpenSearch (log search and analytics)
**Panels**: Individual visualization units within a dashboard:
- **Time Series**: Line/bar charts for metrics over time.
- **Stat**: Single big number — current GPU temp, error rate, queue depth.
- **Table**: Tabular data — top 10 slowest queries, highest-cost models.
- **Heatmap**: Distribution over time — request latency distribution visualized as a heatmap.
- **Logs Panel**: Streaming log viewer filtered by labels.
- **Traces Panel**: Flame graph visualization of distributed traces.
**Dashboards**: Collections of panels arranged on a grid. Shareable as JSON — import community dashboards from grafana.com/grafana/dashboards.
**Alerting**: Grafana Alerting evaluates queries on a schedule and sends notifications via Slack, PagerDuty, email, and webhooks when thresholds are breached.
**Pre-Built AI/ML Dashboards**
| Dashboard | Source | Key Panels |
|-----------|--------|-----------|
| NVIDIA DCGM | grafana.com (ID 12239) | GPU util, temp, memory per device |
| Kubernetes cluster | grafana.com (ID 15661) | Pod health, resource usage |
| vLLM Inference | vLLM docs | TTFT, throughput, queue, KV cache |
| W&B alternative | Custom | Training loss, eval metrics |
| Node Exporter Full | grafana.com (ID 1860) | CPU, memory, disk, network |
**Grafana Stack (LGTM)**
Grafana Labs provides a full open-source observability stack:
- **Loki** — Log aggregation (like Prometheus but for logs).
- **Grafana** — Visualization layer.
- **Tempo** — Distributed tracing backend.
- **Mimir** — Long-term metrics storage (horizontally scalable Prometheus).
Together these four components cover all three observability pillars (metrics, logs, traces) in a single integrated stack.
**Practical AI Inference Dashboard**
A production LLM serving dashboard typically includes:
- TTFT p50/p95/p99 over time (line chart).
- Tokens per second by model (stacked bar).
- Active requests in queue (gauge).
- GPU memory utilization per device (multi-line).
- Error rate by error type (bar chart).
- Cost per 1K tokens trend (time series).
- Top 10 longest prompts by user (table).
Grafana is **the universal lens through which AI teams observe their systems** — its ability to unify metrics, logs, and traces from any data source into a single, interactive view makes it indispensable for monitoring the full stack from GPU hardware to LLM response quality in production.