torchserve
**TorchServe** is a **production-ready serving framework for PyTorch models** — deploying trained models as REST/gRPC services with auto-scaling, batching, and version management for high-performance inference.
**What Is TorchServe?**
- **Purpose**: Serve PyTorch models in production.
- **Deployment**: REST API, gRPC, Docker, Kubernetes.
- **Performance**: Batching, multi-GPU, quantization support.
- **Management**: Model versioning, A/B testing, rolling updates.
- **Scaling**: Horizontal scaling with load balancing.
**Why TorchServe Matters**
- **PyTorch Native**: Built for PyTorch by Meta.
- **High Performance**: Optimized for inference speed.
- **Production Ready**: Built-in monitoring, logging, metrics.
- **Easy Deployment**: Single command deployment.
- **Version Management**: Multiple model versions simultaneously.
- **Community**: Active development, good documentation.
**Key Features**
**Model Management**: Upload, unload, version models.
**Batching**: Automatic batching for throughput.
**Multi-GPU**: Distribute across GPUs.
**Custom Handlers**: Preprocessing, postprocessing logic.
**Metrics**: Prometheus-compatible monitoring.
**Quick Start**
```bash
# Install
pip install torchserve torch-model-archiver
# Create model archive
torch-model-archiver --model-name resnet50 \
--version 1.0 \
--model-file model.py \
--serialized-file resnet50.pt \
--handler image_classifier
# Start TorchServe
torchserve --start --model-store model_store \
--models resnet50=resnet50.mar
# Predict
curl http://localhost:8080/predictions/resnet50 \
-F "[email protected]"
```
**Alternatives**: Seldon, KServe, BentoML, Triton.
TorchServe is the **PyTorch production framework** — deploy models with performance, reliability, scaling.