Model Deployment Serving
# Model Deployment & Serving
## Introduction & Motivation
Model Deployment: put trained models into production. Serving infrastructure; real-time inference. Applications: web services, microservices, batch processing.
Motivation: Operationalize models for end-users.
Applications: Web APIs, mobile apps, batch predictions.
---
## Core Concepts & Theory
### Model Serialization
Save trained models (pkl, SavedModel, ONNX).
### Serving Framework
REST APIs, gRPC services.
### Scalability
Handle concurrent requests.
### Monitoring & Logging
Track performance and errors.
---
## Mathematical Formulation
Throughput:
$$ ext{TPH} = \frac{ ext{requests}}{time}$$
Latency Percentiles:
$$ ext{p95} = 95 ext{-th percentile of latencies}$$
Model Size:
$$ ext{Model Size} = \sum_{ ext{layers}} ext{size}(w) + ext{size}(b)$$
---
## Advanced Theory & Extensions
### Model Versioning
Manage multiple model versions.
### A/B Testing
Compare model versions in production.
### Blue-Green Deployment
Seamless model updates.
---
## Computational Considerations
Inference latency: O(model_complexity).
Memory: O(model_size).
Throughput: O(batch_size).
---
## Practical Implementation Strategies
### Containerization
Docker for consistent deployment.
### Batching
Increase throughput via batch processing.
### Caching
Cache frequent predictions.
---
## Benchmark Datasets & Evaluation
TensorFlow Serving: Real-time inference.
TorchServe: PyTorch model serving.
KServe: Kubernetes-native model serving.
---
## Key Challenges & Limitations
### Latency Requirements
Real-time inference constraints.
### Scalability
Handling millions of requests.
### Model Updates
Seamless redeployment.
---
## Hyperparameter Tuning
Batch size: 1-256 depending on latency/throughput.
Worker threads: 4-32.
Cache size: 1000-100000 entries.
---
## Real-World Applications & Case Studies
Web API: Image classification REST service.
Recommendation System: Real-time personalized suggestions.
NLP Service: Text classification as microservice.
---
## Integration with Other Methods
Model serving + monitoring for model drift detection; + A/B testing for continuous improvement.
---
## Summary & Key Takeaways
Model Deployment via serving frameworks enables production-ready ML systems.
Principles:
1. Serialization: Model persistence.
2. APIs: Request-response interface.
3. Scalability: Concurrent processing.
4. Monitoring: Performance tracking.
5. Versioning: Multiple model management.
---
---
## Appendix: Practical Labs
### Lab 1: Model Serialization
import numpy as np
import pickle
def serialize_model(model_weights, filepath):
"""Save model weights to disk"""
model_dict = {'weights': model_weights}
with open(filepath, 'wb') as f:
pickle.dump(model_dict, f)
def deserialize_model(filepath):
"""Load model weights from disk"""
with open(filepath, 'rb') as f:
model_dict = pickle.load(f)
return model_dict['weights']
# Test
np.random.seed(42)
weights = np.random.randn(100, 50)
serialize_model(weights, '/tmp/test_model.pkl')
loaded = deserialize_model('/tmp/test_model.pkl')
assert np.allclose(weights, loaded), "Model serialization works"
print("✓ Model serialization working")
if __name__ == "__main__":
print("Lab 1: ModelSerialization - PASSED")### Lab 2: Latency Calculation
import numpy as np
def compute_inference_latency(predictions, timestamps):
"""Compute latency statistics"""
latencies = np.diff(timestamps)
mean_latency = np.mean(latencies)
median_latency = np.median(latencies)
p95_latency = np.percentile(latencies, 95)
p99_latency = np.percentile(latencies, 99)
return mean_latency, median_latency, p95_latency, p99_latency
# Test
np.random.seed(42)
timestamps = np.cumsum(np.random.rand(1000) * 0.1)
preds = np.random.rand(1000)
mean, med, p95, p99 = compute_inference_latency(preds, timestamps)
assert mean > 0, "Mean latency positive"
assert p99 >= p95 >= med >= mean, "Latency ordering"
print("✓ Latency calculation working")
if __name__ == "__main__":
print("Lab 2: LatencyCalculation - PASSED")### Lab 3: Request Batching
import numpy as np
class BatchProcessor:
def __init__(self, batch_size=32, timeout=1.0):
self.batch_size = batch_size
self.timeout = timeout
self.queue = []
def add_request(self, request):
"""Add request to batch queue"""
self.queue.append(request)
def should_process(self):
"""Check if batch should be processed"""
return len(self.queue) >= self.batch_size
def get_batch(self):
"""Get current batch"""
batch = self.queue[:self.batch_size]
self.queue = self.queue[self.batch_size:]
return batch
# Test
processor = BatchProcessor(batch_size=4)
for i in range(10):
processor.add_request(i)
batch = processor.get_batch()
assert len(batch) == 4, "Batch size correct"
assert len(processor.queue) == 6, "Remaining queue correct"
print("✓ Request batching working")
if __name__ == "__main__":
print("Lab 3: RequestBatching - PASSED")### Lab 4: Model Versioning
import numpy as np
class ModelVersionManager:
def __init__(self):
self.versions = {}
self.current = None
def register_version(self, version_id, model_weights):
"""Register new model version"""
self.versions[version_id] = model_weights
if self.current is None:
self.current = version_id
def switch_version(self, version_id):
"""Switch to different version"""
if version_id in self.versions:
self.current = version_id
return True
return False
def get_current_model(self):
"""Get current active model"""
return self.versions.get(self.current)
# Test
manager = ModelVersionManager()
v1 = np.random.randn(100)
v2 = np.random.randn(100)
manager.register_version('v1', v1)
manager.register_version('v2', v2)
manager.switch_version('v2')
current = manager.get_current_model()
assert np.allclose(current, v2), "Version switching works"
print("✓ Model versioning working")
if __name__ == "__main__":
print("Lab 4: ModelVersioning - PASSED")