Model Serving Deployment Tensorflow Serving Onnx Production
# Model Serving & Deployment: TensorFlow Serving, ONNX & Production
## Introduction & Motivation
Model serving: deploy trained models for inference. TensorFlow Serving: scalable model serving. ONNX: cross-framework interoperability. Docker: containerization; reproducible deployment. Monitoring: track model performance, data drift. Applications: production ML systems, real-time inference, A/B testing.
Motivation: Training → production gap. Serving infrastructure essential for deployment.
Applications: Web services, real-time systems, batch pipelines.
---
## Core Concepts & Theory
### Containerization
Package model + dependencies; reproducible.
### Versioning
Track model versions; enable rollback.
### Load Balancing
Distribute inference across servers.
---
## Mathematical Formulation
Throughput (queries/second):
$$QPS = \frac{ ext{batch\_size} imes ext{fps}}{1}$$
Latency (ms):
$$L = ext{model\_latency} + ext{queue\_latency}$$
---
## Advanced Theory & Extensions
### Batch Processing
Batch inference; amortize overhead.
### Model Distillation
Smaller model for serving; latency reduction.
### Caching
Cache predictions; avoid redundant computation.
---
## Computational Considerations
Latency: Model forward pass + serving overhead.
Memory: Model size + batch size × input size.
CPU/GPU: Utilization depends on batch size.
---
## Practical Implementation Strategies
### Model Format
SavedModel (TF), ONNX, PyTorch scripted.
### API Design
RESTful (JSON), gRPC (faster).
### Monitoring
Latency, throughput, error rate.
---
## Benchmark Datasets & Evaluation
Latency: <100ms typical for web services.
Throughput: 100-10k QPS depending on model.
Accuracy: Monitor for data drift.
---
## Key Challenges & Limitations
### Latency Requirements
Real-time: <10ms. Web: <100ms.
### Scalability
Load increases; auto-scaling needed.
### Model Updates
Deploy new versions; A/B testing.
---
## Hyperparameter Tuning
Batch size: 1-256; latency-throughput tradeoff.
Number of workers: CPU cores typical.
Timeout: task dependent; typically 30s.
---
## Real-World Applications & Case Studies
TensorFlow Serving: Large-scale TF model serving.
Docker: Standard containerization.
Kubernetes: Orchestration, auto-scaling.
---
## Integration with Other Methods
Serving + Monitoring → production ML ops.
Serving + A/B Testing → online evaluation.
---
## Summary & Key Takeaways
Model serving via containerization, versioning, and load balancing enables production deployment with monitoring for performance and data drift.
Principles:
1. Containerization: Docker reproducible.
2. Versioning: track model evolution.
3. API: RESTful or gRPC.
4. Monitoring: track latency, throughput, accuracy.
5. Scaling: load balancing, auto-scaling.
---
---
## Appendix: Practical Labs
### Lab 1: Model Serialization
import torch
import numpy as np
def save_model_checkpoint(model, optimizer, epoch, path):
"""Save model checkpoint"""
checkpoint = {
'epoch': epoch,
'model_state': model.state_dict(),
'optimizer_state': optimizer.state_dict(),
}
torch.save(checkpoint, path)
def load_model_checkpoint(model, optimizer, path):
"""Load model checkpoint"""
checkpoint = torch.load(path)
model.load_state_dict(checkpoint['model_state'])
optimizer.load_state_dict(checkpoint['optimizer_state'])
epoch = checkpoint['epoch']
return model, optimizer, epoch
# Test
np.random.seed(42)
model = torch.nn.Linear(10, 5)
optimizer = torch.optim.Adam(model.parameters())
save_model_checkpoint(model, optimizer, epoch=10, path='/tmp/ckpt.pt')
loaded_model, loaded_opt, loaded_epoch = load_model_checkpoint(
torch.nn.Linear(10, 5), torch.optim.Adam([torch.nn.Parameter(torch.randn(1))]),
path='/tmp/ckpt.pt'
)
assert loaded_epoch == 10, "Epoch loaded"
print("✓ Model serialization working")
if __name__ == "__main__":
print("Lab 1: Serialization - PASSED")### Lab 2: Batch Inference
import numpy as np
def batch_inference(model, X, batch_size=32):
"""Inference on batches"""
predictions = []
for i in range(0, len(X), batch_size):
batch = X[i:i+batch_size]
pred = model.predict(batch)
predictions.append(pred)
return np.concatenate(predictions)
# Test
np.random.seed(42)
class DummyModel:
def predict(self, x):
return (x ** 2).sum(axis=1)
model = DummyModel()
X = np.random.randn(100, 10)
predictions = batch_inference(model, X, batch_size=16)
assert len(predictions) == 100, "All samples predicted"
print("✓ Batch inference working")
if __name__ == "__main__":
print("Lab 2: BatchInference - PASSED")### Lab 3: Latency Measurement
import time
import numpy as np
def measure_latency(model, X, num_warmup=10, num_runs=100):
"""Measure inference latency"""
# Warmup
for _ in range(num_warmup):
model.predict(X[:1])
# Measure
latencies = []
for _ in range(num_runs):
start = time.time()
model.predict(X[:1])
latency = (time.time() - start) * 1000 # ms
latencies.append(latency)
return {
'mean': np.mean(latencies),
'std': np.std(latencies),
'p99': np.percentile(latencies, 99),
}
# Test
class DummyModel:
def predict(self, x):
time.sleep(0.001)
return np.zeros(len(x))
model = DummyModel()
X = np.random.randn(100, 10)
metrics = measure_latency(model, X, num_warmup=2, num_runs=10)
assert 'mean' in metrics and 'p99' in metrics, "Metrics computed"
print("✓ Latency measurement working")
if __name__ == "__main__":
print("Lab 3: Latency - PASSED")### Lab 4: Throughput Computation
import numpy as np
def compute_throughput(batch_size, latency_ms):
"""Compute queries per second (QPS)"""
# latency in seconds
latency_s = latency_ms / 1000
# Batches per second
batches_per_sec = 1 / latency_s
# Queries per second
qps = batches_per_sec * batch_size
return qps
# Test
batch_size = 32
latency_ms = 100
qps = compute_throughput(batch_size, latency_ms)
assert qps > 0, "QPS positive"
assert qps == batch_size / 0.1, "QPS calculation correct"
print("✓ Throughput computation working")
if __name__ == "__main__":
print("Lab 4: Throughput - PASSED")