serverless
**Serverless ML Inference**
**When to Use Serverless**
| Factor | Serverless Good | Serverless Bad |
|--------|-----------------|----------------|
| Traffic | Sporadic/variable | Constant high |
| Cold start | Acceptable | Not acceptable |
| Model size | Small (<10GB) | Large (>10GB) |
| Latency | Seconds OK | Milliseconds needed |
| Cost at scale | Higher | Lower |
**AWS Lambda for ML**
```python
# handler.py
import json
import boto3
from transformers import pipeline
# Load model outside handler (reused across invocations)
classifier = pipeline("sentiment-analysis", model="distilbert-base-uncased")
def handler(event, context):
text = event.get("text", "")
result = classifier(text)
return {
"statusCode": 200,
"body": json.dumps(result)
}
```
**Lambda Container Images**
```dockerfile
FROM public.ecr.aws/lambda/python:3.11
# Install dependencies
COPY requirements.txt .
RUN pip install -r requirements.txt
# Copy model and code
COPY model/ /var/task/model/
COPY handler.py .
CMD ["handler.handler"]
```
**Lambda Limitations**
| Limit | Value |
|-------|-------|
| Memory | 10 GB max |
| Timeout | 15 minutes |
| Package size | 10 GB (container) |
| Temp storage | 10 GB /tmp |
| Concurrent | 1000 default (adjustable) |
**AWS SageMaker Serverless**
```python
import sagemaker
# Create serverless inference endpoint
predictor = model.deploy(
serverless_inference_config=ServerlessInferenceConfig(
memory_size_in_mb=4096,
max_concurrency=10,
)
)
```
**Google Cloud Functions / Cloud Run**
```python
# Cloud Run (container-based, larger models)
from flask import Flask, request
app = Flask(__name__)
model = load_model()
@app.route("/predict", methods=["POST"])
def predict():
data = request.json
result = model.predict(data["input"])
return {"prediction": result}
```
**Cold Start Mitigation**
| Strategy | AWS | GCP |
|----------|-----|-----|
| Provisioned | Provisioned Concurrency | Min instances |
| Warming | CloudWatch scheduled | Cloud Scheduler |
| Container | Keep container warm | Cloud Run min instances |
**Cost Comparison**
```
Lambda (1M requests, 1s each, 4GB memory):
- $0.0000066667 per GB-second
- Cost: ~$27/month
EC2 g4dn.xlarge (always on):
- $0.526/hr = $383/month
Break-even: ~1.4M seconds of compute/month
```
**Best Practices**
- Keep models small or use model-as-service
- Use container images for larger dependencies
- Provision concurrency for low latency
- Consider Cloud Run for larger models
- Monitor cold start metrics