Infrastructure as Code for ML

Why IaC for ML? Reproducible, version-controlled infrastructure for ML pipelines, training clusters, and inference services.

Terraform Basics

# Provider configuration
provider "aws" {
  region = "us-east-1"
}

# GPU instance for inference
resource "aws_instance" "llm_server" {
  ami           = "ami-xxx"  # Deep Learning AMI
  instance_type = "g4dn.xlarge"

  tags = {
    Name = "llm-inference-server"
  }
}

# S3 bucket for models
resource "aws_s3_bucket" "models" {
  bucket = "company-llm-models"
}

EKS Cluster for ML

module "eks" {
  source  = "terraform-aws-modules/eks/aws"

  cluster_name    = "ml-cluster"
  cluster_version = "1.28"

  node_groups = {
    cpu = {
      instance_types = ["m5.2xlarge"]
      capacity_type  = "ON_DEMAND"
      desired_size   = 3
    }

    gpu = {
      instance_types = ["g4dn.xlarge"]
      capacity_type  = "SPOT"
      desired_size   = 2

      labels = {
        "nvidia.com/gpu" = "true"
      }

      taints = [{
        key    = "nvidia.com/gpu"
        value  = "true"
        effect = "NO_SCHEDULE"
      }]
    }
  }
}

Model Serving Infrastructure

# Load balancer
resource "aws_lb" "llm_api" {
  name               = "llm-api-lb"
  load_balancer_type = "application"
  subnets            = var.public_subnets
}

# Auto-scaling group
resource "aws_autoscaling_group" "llm_servers" {
  desired_capacity = 2
  max_size         = 10
  min_size         = 1

  launch_template {
    id      = aws_launch_template.llm_server.id
    version = "$Latest"
  }

  # Scale based on GPU utilization
  target_tracking_configuration {
    customized_metric_specification {
      metric_name = "GPUUtilization"
      namespace   = "Custom/ML"
      statistic   = "Average"
    }
    target_value = 70.0
  }
}

Pulumi (Python IaC)

import pulumi
import pulumi_aws as aws

# GPU instance
llm_server = aws.ec2.Instance("llm-server",
    instance_type="g4dn.xlarge",
    ami="ami-xxx",
    tags={"Name": "llm-inference"}
)

# Export endpoint
pulumi.export("server_ip", llm_server.public_ip)

ML-Specific Resources

Best Practices

• Use modules for reusable components
• Separate environments (dev/staging/prod)
• Store state remotely (S3, Terraform Cloud)
• Use variables for configuration
• Tag resources for cost tracking