GPU Management in Kubernetes

NVIDIA GPU Operator Automates GPU driver installation, container toolkit, and device plugins in Kubernetes.

Installation

# Add NVIDIA Helm repo
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia

# Install GPU Operator
helm install gpu-operator nvidia/gpu-operator
  --namespace gpu-operator --create-namespace

Components Installed

Requesting GPUs in Pods

apiVersion: v1
kind: Pod
metadata:
  name: llm-server
spec:
  containers:
  - name: inference
    image: llm-inference:latest
    resources:
      limits:
        nvidia.com/gpu: 1  # Request 1 GPU

Multiple GPUs

resources:
  limits:
    nvidia.com/gpu: 4  # Multi-GPU for large models

Node Selectors for GPU Types

spec:
  nodeSelector:
    nvidia.com/gpu.product: "NVIDIA-A100-SXM4-80GB"
  containers:
  - name: model
    resources:
      limits:
        nvidia.com/gpu: 1

GPU Sharing (Time-Slicing)

# ConfigMap for time-slicing
apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
data:
  any: |-
    version: v1
    sharing:
      timeSlicing:
        resources:
        - name: nvidia.com/gpu
          replicas: 4  # 4 pods can share each GPU

MIG (Multi-Instance GPU) Split A100/H100 into multiple instances:

resources:
  limits:
    nvidia.com/mig-3g.20gb: 1  # 3GB compute, 20GB memory slice

Monitoring GPUs

# Check GPU allocation
kubectl describe nodes | grep nvidia.com/gpu

# View GPU metrics
kubectl logs -n gpu-operator dcgm-exporter-xxx

# GPU utilization in Grafana via DCGM metrics

Best Practices

• Use GPU Operator for consistent setup
• Set appropriate GPU limits
• Use node selectors for GPU types
• Monitor GPU utilization
• Consider time-slicing for dev environments
• Use MIG for flexible resource allocation