Cortex: Serverless ML Infrastructure

Overview Cortex is an open-source platform for deploying machine learning models as production-ready web APIs. It automates the infrastructure underlying model serving on AWS (EC2, EKS), abstracting away Kubernetes and Docker complexity.

Key Features

1. Unified Config Deploy TensorFlow, PyTorch, Scikit-learn, or ONNX models using a simple cortex.yaml file.

2. Autoscaling Automatically scales the number of replicas based on request traffic (Requests Per Second) or GPU utilization. Scales to zero to save costs.

3. Spot Instances Built-in support for AWS Spot Instances, potentially saving 70-90% on compute costs, with auto-recovery if instances are reclaimed.

4. Rolling Updates Updates APIs without downtime.

Configuration Example

# cortex.yaml
- name: sentiment-analyzer
  kind: RealtimeAPI
  predictor:
    type: python
    path: predictor.py
  compute:
    cpu: 1
    gpu: 1  # Uses GPU instance
    mem: 4G
  autoscaling:
    min_replicas: 1
    max_replicas: 10

Python Predictor

# predictor.py
class PythonPredictor:
    def __init__(self, config):
        self.model = load_model()

    def predict(self, payload):
        return self.model.inference(payload["text"])

Status Note: Cortex was acquired by Databricks, and the open-source project is no longer actively maintained as of late 2021/2022. Modern Alternatives include:

• BentoML: For packaging models.
• Ray Serve: For scalable serving.
• KServe: For Kubernetes native serving.
• AWS SageMaker: Managed alternative.