kserve
**KServe** is a **standardized, serverless inference platform for Kubernetes that provides scale-to-zero autoscaling, canary rollouts, and a unified V2 inference protocol** — originally developed as KFServing by Google, IBM, and Bloomberg, KServe builds on Knative and Istio to automatically manage the full lifecycle of model serving on Kubernetes, including traffic routing between model versions, pre/post-processing transformers, GPU autoscaling, and support for every major serving runtime (TF Serving, TorchServe, Triton, MLServer).
**What Is KServe?**
- **Definition**: A Kubernetes-native platform (formerly KFServing) for deploying, managing, and scaling ML inference services, providing a custom resource definition (InferenceService) that abstracts away the complexity of Kubernetes networking, autoscaling, and traffic management.
- **The Problem**: Deploying one model on Kubernetes is manageable. Deploying 500 models with different frameworks, GPU requirements, traffic patterns, and version rollout strategies is a nightmare of YAML engineering. KServe automates all of this.
- **Scale-to-Zero**: Unlike always-on deployments, KServe scales model pods down to zero when there's no traffic — and scales back up automatically when a request arrives. This can reduce costs by 80%+ for infrequently-used models.
**Core Features**
| Feature | Description | Benefit |
|---------|------------|---------|
| **Scale-to-Zero** | Pods spin down when idle, spin up on request | 80%+ cost savings on low-traffic models |
| **Canary Rollouts** | Route 10% traffic to v2, 90% to v1 | Safe production deployments |
| **V2 Protocol** | Standardized inference API across all runtimes | Framework-agnostic client code |
| **Transformers** | Pre/post-processing as separate containers | Separation of concerns (data prep vs inference) |
| **Model Mesh** | Multi-model serving for high-density deployments | Serve 1000s of models on shared infrastructure |
| **GPU Autoscaling** | Scale GPU pods based on queue depth/latency | Cost-efficient GPU utilization |
**Supported Serving Runtimes**
| Runtime | Framework | Notes |
|---------|-----------|-------|
| **TF Serving** | TensorFlow SavedModel | Google's production server |
| **TorchServe** | PyTorch | PyTorch's official server |
| **Triton** | TF, PyTorch, ONNX, TensorRT | NVIDIA's multi-framework server |
| **MLServer** | Scikit-Learn, XGBoost, LightGBM | Seldon's Python server |
| **Custom** | Any container with HTTP endpoint | Maximum flexibility |
**KServe Architecture**
| Component | Role |
|-----------|------|
| **Predictor** | The main model serving container |
| **Transformer** | Pre-processing (tokenization, image resize) and post-processing (label mapping) |
| **Explainer** | Model interpretability (SHAP, LIME) served alongside predictions |
| **Knative** | Provides serverless autoscaling (scale-to-zero) |
| **Istio** | Handles traffic routing (canary splits, A/B testing) |
**KServe vs Alternatives**
| Feature | KServe | Seldon Core | SageMaker Endpoints | BentoML |
|---------|--------|------------|-------------------|---------|
| **Platform** | Kubernetes (any cloud) | Kubernetes | AWS only | Any (BentoCloud optional) |
| **Scale-to-Zero** | Yes (Knative) | No (always-on) | No | Yes (Bento Cloud) |
| **Multi-Framework** | Yes (pluggable runtimes) | Yes | Yes | Yes |
| **Canary Rollouts** | Yes (Istio) | Yes (Istio) | Yes (native) | Manual |
| **Best For** | K8s-native teams, multi-cloud | Enterprise K8s deployments | AWS-only shops | Rapid deployment |
**KServe is the standard serverless inference platform for Kubernetes** — providing scale-to-zero autoscaling, seamless canary rollouts, and a unified V2 inference protocol that abstracts away Kubernetes complexity while supporting every major serving runtime, making it the production choice for organizations running hundreds of ML models at scale on Kubernetes.