gpu mps
**GPU Multi-Process Service (MPS)** is the **NVIDIA runtime service that enables multiple CUDA processes to share a single GPU concurrently with improved efficiency** — replacing the default time-slicing behavior (where processes alternate GPU access) with true spatial sharing where multiple processes' kernels execute simultaneously on the same GPU, improving utilization for workloads like multi-rank MPI jobs, inference serving with multiple workers, and Kubernetes GPU sharing.
**Why MPS**
- Default GPU sharing: Time-slicing via context switching → only one process uses GPU at a time.
- Context switch cost: ~25-50 µs → each process gets exclusive GPU access for a time quantum.
- Problem: Small kernels from one process don't fill the GPU → 30-50% utilization waste.
- MPS: Funnel all processes through a single CUDA context → kernels from different processes run simultaneously.
**How MPS Works**
```
Without MPS (time-slicing):
Process A: [kernel][idle ][kernel][idle ]
Process B: [idle ][kernel][idle ][kernel]
GPU: [ A ][ B ][ A ][ B ] ← context switches
With MPS:
Process A: [kernel][kernel][kernel]
Process B: [kernel][kernel][kernel]
GPU: [ A+B ][ A+B ][ A+B ] ← concurrent execution
```
**Starting MPS**
```bash
# Start MPS daemon (run as root or GPU owner)
export CUDA_VISIBLE_DEVICES=0
nvidia-cuda-mps-control -d
# All CUDA processes on GPU 0 now go through MPS
# Run multiple processes
mpirun -np 4 ./my_cuda_app # 4 MPI ranks share GPU via MPS
# Stop MPS
echo quit | nvidia-cuda-mps-control
```
**MPS Benefits**
| Scenario | Without MPS | With MPS | Improvement |
|----------|------------|----------|-------------|
| 4 MPI ranks, small kernels | 35% GPU util | 85% GPU util | 2.4× |
| 8 inference workers | 40% GPU util | 90% GPU util | 2.3× |
| Context switch overhead | 25-50 µs/switch | 0 (shared context) | Eliminated |
| Memory overhead | N contexts × overhead | 1 shared context | Reduced |
**MPS vs. MIG vs. Time-Slicing**
| Feature | Time-Slicing | MPS | MIG |
|---------|-------------|-----|-----|
| Isolation | Temporal only | Minimal | Full hardware |
| Concurrent execution | No | Yes | Yes (separate instances) |
| Memory protection | Full | Limited | Full |
| Error isolation | Full | Shared (one crash affects all) | Full |
| Overhead | Context switch | Minimal | Partitioning setup |
| GPU support | All | Volta+ | A100+ |
| Best for | Mixed workloads | MPI, cooperative processes | Multi-tenant, cloud |
**Resource Limits (Volta+)**
```bash
# Limit each MPS client to 25% of GPU threads
export CUDA_MPS_ACTIVE_THREAD_PERCENTAGE=25
# With Volta MPS: Up to 48 clients per GPU
# Each client gets guaranteed thread allocation
```
**Use Cases**
- **MPI + GPU**: 4-8 MPI ranks per GPU → each rank launches small kernels → MPS packs them together.
- **Inference serving**: Multiple model workers share one GPU → reduce cost per query.
- **Kubernetes**: GPU sharing without MIG hardware support → MPS as lightweight alternative.
- **Hyperparameter search**: Multiple small training runs share GPU resources.
**Limitations**
- No memory protection between clients → one process can corrupt another's data.
- One client failure can crash all MPS clients on that GPU.
- Unified memory not fully supported with MPS.
- Cannot mix MPS and non-MPS processes on the same GPU.
GPU Multi-Process Service is **the lightweight GPU sharing solution for cooperative workloads** — by eliminating context switching and enabling true spatial multiplexing of multiple CUDA processes on a single GPU, MPS transforms underutilized GPUs running many small tasks into efficiently packed compute resources, making it essential for MPI-based HPC applications and cost-effective inference serving where workloads are trusted and isolation requirements are relaxed.