cog
**Cog** is an **open-source tool by Replicate that packages machine learning models into standard, production-ready Docker containers** — solving the "works on my machine" problem by using a simple cog.yaml configuration file to automatically generate Dockerfiles with correct CUDA drivers, Python versions, system dependencies, and a standardized HTTP prediction API, turning any Python model into a deployable container without writing a single line of Docker configuration.
**What Is Cog?**
- **Definition**: A command-line tool (pip install cog) that takes a Python prediction class and a YAML configuration file and produces a fully functional Docker container with an HTTP API at /predictions — handling all the CUDA, system library, and Python dependency complexity automatically.
- **The Problem**: Data scientists train models in Jupyter notebooks with a chaotic mix of pip, conda, system packages, and specific CUDA versions. Getting this into a Docker container requires deep DevOps knowledge — writing Dockerfiles, managing CUDA driver compatibility, setting up HTTP endpoints, and handling GPU memory.
- **The Solution**: Define dependencies in cog.yaml, write a predict() function, run `cog build` — done. Cog generates the Dockerfile, builds the container, and provides a standardized API.
**How Cog Works**
| Step | What You Do | What Cog Does |
|------|------------|--------------|
| 1. Define dependencies | Write cog.yaml with Python version + packages | Generates multi-stage Dockerfile |
| 2. Write predict function | Python class with setup() and predict() methods | Creates HTTP /predictions endpoint |
| 3. Build | Run `cog build` | Builds Docker image with CUDA, dependencies |
| 4. Test locally | Run `cog predict -i [email protected]` | Runs prediction in container |
| 5. Deploy | Push to Replicate or any Docker host | Instant API hosting |
**cog.yaml Example**
```yaml
build:
gpu: true
python_version: "3.10"
python_packages:
- torch==2.1
- transformers==4.36
system_packages:
- ffmpeg
predict: "predict.py:Predictor"
```
**predict.py Example**
```python
from cog import BasePredictor, Input, Path
class Predictor(BasePredictor):
def setup(self):
"""Load model into memory (runs once on startup)"""
self.model = load_model("weights/model.pt")
def predict(self, image: Path = Input(description="Input image")) -> Path:
"""Run inference on an input image"""
output = self.model(image)
return Path(output)
```
**Cog vs Alternatives**
| Tool | Approach | Strengths | Limitations |
|------|---------|-----------|-------------|
| **Cog** | YAML + predict class → Docker | Simplest path to container, Replicate integration | Replicate-specific ecosystem |
| **BentoML** | Python decorators → Bento → container | More flexible, multi-model support | More complex API |
| **Docker (manual)** | Write Dockerfile from scratch | Full control | Requires Docker expertise, CUDA pain |
| **TorchServe / TF Serving** | Framework-specific server | Optimized for specific framework | Framework lock-in |
| **Triton** | NVIDIA inference server | Best GPU performance | Complex configuration |
**Cog is the fastest path from ML model to production Docker container** — eliminating the DevOps complexity of CUDA drivers, system dependencies, and HTTP API setup through a simple YAML configuration and Python prediction class, enabling data scientists to package any model into a standardized, deployable container without Docker expertise.