gpu multi instance gpu mig

**GPU Multi-Instance GPU (MIG)** is **a hardware partitioning feature introduced with NVIDIA's A100 (Ampere) architecture that divides a single physical GPU into up to seven independent instances, each with dedicated compute resources, memory bandwidth, and memory capacity** — MIG enables multiple users or workloads to share a GPU with hardware-level isolation, guaranteed quality of service, and no performance interference. **MIG Architecture:** - **GPU Instances (GI)**: the first level of partitioning divides the GPU's streaming multiprocessors (SMs) and memory into isolated GPU Instances — each GI has its own memory partition and dedicated portion of the L2 cache - **Compute Instances (CI)**: each GPU Instance can be further subdivided into Compute Instances that share the GI's memory but have dedicated SM resources — enables finer-grained compute partitioning within a memory domain - **Hardware Isolation**: MIG uses hardware memory firewalls between instances — one instance cannot access another's memory, providing security isolation equivalent to separate physical GPUs - **Fault Isolation**: ECC errors, GPU hangs, or crashes in one MIG instance don't affect other instances — each instance operates as an independent GPU with its own error handling **A100 MIG Configurations:** - **Full GPU**: 108 SMs, 80 GB HBM2e, 2039 GB/s bandwidth — used when a single workload needs maximum resources - **7× 1g.5gb**: seven instances with ~14 SMs and ~5 GB each — maximum multi-tenancy for small inference workloads - **3× 2g.10gb + 1× 1g.5gb**: three medium instances plus one small — mixed workload deployment - **2× 3g.20gb + 1× 1g.5gb**: two larger instances plus one small — balanced compute and memory for moderate workloads - **1× 4g.20gb + 1× 3g.20gb**: two large instances — suitable for two concurrent training jobs or large inference models **MIG Setup and Management:** - **Enable MIG Mode**: nvidia-smi -i 0 --mig-enabled — requires GPU reset, sets the GPU into MIG-capable mode (driver support required) - **Create GPU Instance**: nvidia-smi mig -i 0 -cgi 9,3,3 — creates one 4g.20gb (profile 9) and two 2g.10gb (profile 3) GPU Instances - **Create Compute Instance**: nvidia-smi mig -i 0 -gi 0 -cci 0 — creates a Compute Instance within GPU Instance 0, making it usable by applications - **Device Enumeration**: CUDA_VISIBLE_DEVICES=MIG-GPU-// selects a specific MIG instance — applications see it as a standalone GPU with no awareness of MIG partitioning **Use Cases and Deployment:** - **Multi-Tenant Inference**: cloud providers assign MIG instances to different customers — each customer gets guaranteed GPU resources without noisy-neighbor interference, improving SLA compliance - **Development and Testing**: developers share a single A100 by each receiving a MIG slice — 7 developers can simultaneously develop and test GPU code on one physical GPU - **Mixed Workload Consolidation**: run inference serving on smaller slices while a training job uses a larger slice — improves overall GPU utilization from typical 30-40% to 80-90% - **Kubernetes Integration**: NVIDIA's device plugin exposes MIG instances as individual GPU resources — Kubernetes schedules pods to specific MIG slices using standard resource requests **Performance Characteristics:** - **Linear Scaling**: a 1g.5gb instance provides approximately 1/7 of full GPU compute, a 3g.20gb provides approximately 3/7 — performance scales linearly with allocated SM count for compute-bound workloads - **Memory Bandwidth**: each instance gets a proportional share of HBM bandwidth — a 2g.10gb instance receives approximately 2/7 of total bandwidth, sufficient for many inference workloads - **L2 Cache Partitioning**: the L2 cache is physically partitioned between instances — no cache interference means predictable performance regardless of co-running workloads - **No Oversubscription**: MIG doesn't allow allocating more resources than physically available — unlike time-slicing (MPS), MIG provides hard resource boundaries **Comparison with Other GPU Sharing:** - **MPS (Multi-Process Service)**: time-shares SM resources without memory isolation — higher utilization for cooperative workloads but no QoS guarantees or security isolation - **Time-Slicing (vGPU)**: context-switches the entire GPU between users — provides isolation but serializes execution, Adding latency jitter - **MIG Advantage**: only approach providing simultaneous execution with hardware isolation — combines the utilization benefits of MPS with the isolation guarantees of separate GPUs **MIG has fundamentally changed GPU datacenter economics — by enabling safe multi-tenancy with hardware-enforced isolation, a single A100 can serve 7 independent inference workloads simultaneously, reducing per-workload GPU cost by up to 7× while maintaining predictable performance.**