gpu multi tenancy
**GPU Multi-Tenancy** is the **sharing of a single physical GPU among multiple applications, users, or virtual machines**, providing isolation, fairness, and efficient utilization of expensive GPU resources that would otherwise sit idle when any single workload cannot fully saturate the device.
GPUs are expensive ($10,000-$40,000+ for data center GPUs) yet many workloads — inference serving, interactive development, small training jobs — utilize only 10-30% of GPU capacity. Multi-tenancy enables cost-effective GPU sharing, which is critical for cloud providers and enterprise GPU clusters.
**GPU Sharing Mechanisms**:
| Mechanism | Isolation | Granularity | Overhead | Vendor |
|-----------|----------|------------|---------|--------|
| **Time-slicing** | Temporal | Full GPU, interleaved | Context switch ~25us | All |
| **MPS** (Multi-Process Service) | Spatial (partial) | SM partitioning | Minimal | NVIDIA |
| **MIG** (Multi-Instance GPU) | Hardware | Fixed GPU fractions | None | NVIDIA A100+ |
| **SR-IOV** | Hardware (VM) | Virtual functions | Low | AMD, Intel |
| **vGPU** (mediated pass-through) | Software | Virtual GPU profiles | Medium | NVIDIA, AMD |
**Time-Slicing**: The GPU scheduler context-switches between multiple applications, giving each a time quantum of full GPU access. Simple and universally available. Drawbacks: context switch overhead (~25 microseconds on modern GPUs), no memory isolation (potential interference), and bursty latency (applications wait their turn). Suitable for development and non-latency-sensitive workloads.
**NVIDIA MPS (Multi-Process Service)**: A daemon that funnels multiple CUDA contexts through a single hardware context, enabling true spatial sharing where multiple processes' kernels execute concurrently on different SMs. Benefits: eliminates context switching overhead, enables fine-grained SM sharing, and supports CUDA streams from different processes. Limitations: limited error isolation (one process faulting affects others), no memory protection between processes, and fixed partitioning of SM resources.
**MIG (Multi-Instance GPU)**: Available on NVIDIA A100, A30, H100. Hardware-level partitioning divides the GPU into up to 7 independent instances, each with dedicated SMs, memory, and L2 cache. Full hardware isolation — one instance's fault or performance behavior doesn't affect others. Each MIG instance appears as an independent GPU to software. Limitation: partition sizes are predefined (not arbitrary), and total partitions are limited.
**Kubernetes GPU Scheduling**: For GPU clusters, resource management integrates with orchestration: **NVIDIA Device Plugin** exposes GPUs as schedulable Kubernetes resources; **GPU sharing extensions** enable fractional GPU allocation (e.g., 0.5 GPU); **topology-aware scheduling** considers NVLink topology and NUMA affinity; **priority-based preemption** enables high-priority workloads to preempt low-priority GPU tenants.
**Fairness and QoS**: Multi-tenant GPU scheduling must ensure: **fair share** (each tenant receives proportional GPU time), **latency SLO** (inference workloads need bounded response time), **memory isolation** (one tenant cannot access or corrupt another's data), and **admission control** (reject workloads that would degrade existing tenants below their SLOs).
**GPU multi-tenancy is transforming GPUs from dedicated single-user devices into shared infrastructure resources — enabling cloud-scale GPU economics where utilization approaching CPU-level sharing efficiency unlocks the full value of expensive accelerator hardware.**