kubernetes for ml

**Kubernetes (K8s) for ML** is the practice of using the Kubernetes container orchestration platform to **deploy, scale, manage, and operate** machine learning workloads — including model training, inference serving, data pipelines, and experiment tracking. **Why Kubernetes for ML** - **GPU Scheduling**: Kubernetes natively supports GPU resource requests and limits, enabling efficient GPU sharing and allocation across ML workloads. - **Auto-Scaling**: Horizontal Pod Autoscaler (HPA) and custom metrics enable automatic scaling of inference services based on demand. - **Reproducibility**: Container images ensure consistent environments across development, testing, and production. - **Multi-Tenancy**: Multiple teams can share a GPU cluster with resource quotas and namespace isolation. **Key K8s Components for ML** - **GPU Device Plugin**: Exposes GPU resources to the Kubernetes scheduler. Supports NVIDIA, AMD, and Intel GPUs. - **Node Selectors / Taints**: Direct ML workloads to GPU-equipped nodes while keeping CPU workloads on cheaper nodes. - **Persistent Volumes**: Store training data, model checkpoints, and datasets on persistent storage that survives pod restarts. - **Jobs / CronJobs**: Run training jobs, batch inference, and scheduled data pipelines. **ML-Specific Kubernetes Tools** - **Kubeflow**: End-to-end ML platform on K8s — provides pipelines, experimentation, serving, and notebook management. - **KServe (formerly KFServing)**: Kubernetes-native model serving with auto-scaling, canary deployments, and multi-model serving. - **Ray on K8s (KubeRay)**: Run distributed training and inference with Ray on Kubernetes. - **Seldon Core**: Advanced model serving with A/B testing, explanations, and drift detection. - **Volcano**: Batch scheduling for Kubernetes optimized for ML and HPC workloads. - **NVIDIA GPU Operator**: Automates GPU driver management, monitoring, and device plugin deployment. **Deployment Patterns** - **Model Serving**: Deploy inference servers (vLLM, TGI, Triton) as Kubernetes Deployments with GPU requests and HPA. - **Training Jobs**: Submit distributed training as Kubernetes Jobs using PyTorchJob or MPIJob custom resources. - **A/B Testing**: Use Istio or KServe traffic splitting to route percentages of traffic to different model versions. Kubernetes is the **de facto standard** infrastructure for production ML at scale — most production LLM deployments run on Kubernetes clusters.