Docker and Kubernetes for ML provide containerization and orchestration infrastructure for deploying machine learning models at scale — packaging models with dependencies into portable containers and managing clusters of GPU-enabled nodes for production serving, training jobs, and auto-scaling inference workloads.
Why Containers for ML?
- Reproducibility: Same environment everywhere (dev, test, prod).
- Dependency Isolation: No conflicts between project requirements.
- Portability: Run anywhere containers run.
- Scaling: Deploy multiple instances easily.
- GPU Support: NVIDIA Container Toolkit enables GPU access.
Docker Basics for ML
Basic Dockerfile:
``dockerfile
FROM nvidia/cuda:12.1-runtime-ubuntu22.04
# Install Python
RUN apt-get update && apt-get install -y python3 python3-pip
# Install dependencies
COPY requirements.txt .
RUN pip3 install -r requirements.txt
# Copy application code
COPY . /app
WORKDIR /app
# Run inference server
CMD ["python3", "serve.py"]
`
Optimized Multi-Stage Build:
`dockerfile
# Build stage
FROM python:3.10-slim AS builder
COPY requirements.txt .
RUN pip install --user -r requirements.txt
# Runtime stage
FROM nvidia/cuda:12.1-runtime-ubuntu22.04
COPY --from=builder /root/.local /root/.local
COPY . /app
WORKDIR /app
ENV PATH=/root/.local/bin:$PATH
CMD ["python", "serve.py"]
`
GPU in Docker:
`bash
# Install NVIDIA Container Toolkit
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
# Run with GPU access
docker run --gpus all -it my-ml-image
# Specific GPUs
docker run --gpus device=0,1 -it my-ml-image
`
Docker Compose for ML:
`yaml`
version: "3.8"
services:
inference:
build: .
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
ports:
- "8000:8000"
volumes:
- ./models:/app/models
environment:
- MODEL_PATH=/app/models/model.pt
Kubernetes for ML
Why Kubernetes?:
- Scale inference across many nodes.
- Manage GPU allocation automatically.
- Self-healing: restart failed pods.
- Load balancing across replicas.
- Rolling updates without downtime.
Deployment Example:
`yaml`
apiVersion: apps/v1
kind: Deployment
metadata:
name: llm-inference
spec:
replicas: 3
selector:
matchLabels:
app: llm-inference
template:
metadata:
labels:
app: llm-inference
spec:
containers:
- name: inference
image: my-registry/llm-server:v1
resources:
limits:
nvidia.com/gpu: 1
ports:
- containerPort: 8000
readinessProbe:
httpGet:
path: /health
port: 8000
Service & Load Balancing:
`yaml`
apiVersion: v1
kind: Service
metadata:
name: llm-service
spec:
selector:
app: llm-inference
ports:
- port: 80
targetPort: 8000
type: LoadBalancer
Horizontal Pod Autoscaler:
`yaml`
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: llm-inference-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: llm-inference
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
ML Platforms on Kubernetes
`
Platform | Purpose | Use Case
-----------|----------------------------|-----------------------
KServe | Model serving | Deploy models easily
Kubeflow | Full ML pipeline | Training + serving
Ray | Distributed compute | Large-scale training
Seldon | ML deployment platform | Enterprise serving
MLflow | Experiment tracking | Model versioning
Best Practices
Container Best Practices:
- Use specific version tags, not :latest.
- Multi-stage builds to reduce image size.
- Don't include training data in images.
- Use .dockerignore to exclude unnecessary files.
- Health checks for readiness/liveness.
K8s Best Practices:
- Set resource requests AND limits.
- Use NVIDIA device plugin for GPU scheduling.
- Implement graceful shutdown for model unloading.
- Use PersistentVolumes for model storage.
- Monitor GPU memory usage.
Docker and Kubernetes are the production backbone of ML infrastructure — enabling reproducible deployments, horizontal scaling, and robust operations that transform ML experiments into reliable production systems.