Deployment Strategies for ML Models

Deployment Strategies

Blue-Green Deployment Two identical environments, switch traffic instantly:

                 [Load Balancer]
                       |
           +-----------+-----------+
           |                       |
    [Blue (current)]        [Green (new)]
           |                       |
        active               preparing

# Blue active, deploy to Green
kubectl apply -f green-deployment.yaml

# Verify Green is healthy
kubectl wait --for=condition=ready pod -l app=green

# Switch traffic
kubectl patch service llm-service -p '{"spec":{"selector":{"version":"green"}}}'

Canary Deployment Gradual traffic shift:

# Nginx Ingress Canary
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: llm-canary
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10"  # 10% to canary
spec:
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /v1/completions
        backend:
          service:
            name: llm-canary
            port: 80

A/B Testing Route by user attributes:

def route_request(request, user_id):
    # Hash user to consistent bucket
    bucket = hash(user_id) % 100

    if bucket < 10:  # 10% to new model
        return call_model_v2(request)
    else:
        return call_model_v1(request)

ML Model Rollout

# Argo Rollouts example
apiVersion: argoproj.io/v1alpha1
kind: Rollout
spec:
  strategy:
    canary:
      steps:
      - setWeight: 5
      - pause: {duration: 10m}
      - setWeight: 25
      - pause: {duration: 10m}
      - setWeight: 50
      - pause: {duration: 10m}
      - setWeight: 100
      analysis:
        templates:
        - templateName: success-rate

Comparison

ML-Specific Concerns

Best Practices

• Always have rollback plan
• Monitor model quality metrics during rollout
• Use canary for high-risk changes
• Automate deployment pipeline