Container Registries for ML

Why Container Registries? Store and deploy ML model containers with versioning, security scanning, and access control.

Major Registries

ECR Setup

# Create repository
aws ecr create-repository --repository-name llm-inference

# Authenticate Docker
aws ecr get-login-password | docker login --username AWS --password-stdin
    123456789.dkr.ecr.us-east-1.amazonaws.com

# Build and push
docker build -t llm-inference .
docker tag llm-inference:latest 123456789.dkr.ecr.us-east-1.amazonaws.com/llm-inference:v1
docker push 123456789.dkr.ecr.us-east-1.amazonaws.com/llm-inference:v1

Image Tagging Strategy

# Tag by version
llm-inference:1.0.0
llm-inference:1.0.1

# Tag by git commit
llm-inference:abc1234

# Tag by model version
llm-inference:gpt4-v2

# Tag by date
llm-inference:2024-01-15

ML-Specific Considerations

Dockerfile for ML

# Multi-stage build
FROM python:3.11-slim as builder
COPY requirements.txt .
RUN pip wheel --no-cache-dir --wheel-dir=/wheels -r requirements.txt

FROM nvidia/cuda:12.1-runtime-ubuntu22.04
COPY --from=builder /wheels /wheels
RUN pip install --no-cache /wheels/*

COPY app/ /app/
WORKDIR /app

# Dont include model weights in image
# Mount from S3 or volume at runtime
ENTRYPOINT ["python", "serve.py"]

Kubernetes ImagePullPolicy

spec:
  containers:
  - name: llm-server
    image: 123456.dkr.ecr.us-east-1.amazonaws.com/llm-inference:v1.2.0
    imagePullPolicy: IfNotPresent  # Cache locally

Best Practices

• Use immutable tags (version, not :latest)
• Enable vulnerability scanning
• Clean up old images (lifecycle policies)
• Use multi-stage builds for smaller images
• Store model weights separately from code