auto-scaling

**Auto-scaling** is the capability to **automatically adjust** the number of compute resources (instances, containers, GPUs) allocated to a service based on real-time demand. It ensures that AI systems have enough capacity during peak loads while minimizing costs during low-traffic periods. **How Auto-Scaling Works** - **Monitoring**: Continuously track metrics like CPU usage, GPU utilization, request queue depth, latency, or token throughput. - **Scaling Policy**: Define rules that trigger scaling actions — e.g., "add 2 instances when average GPU utilization exceeds 80% for 5 minutes." - **Scale Out**: When demand increases, automatically launch new instances to handle the load. - **Scale In**: When demand decreases, automatically terminate excess instances to reduce costs. - **Cooldown Period**: Wait a defined period after a scaling action before evaluating again to prevent oscillation. **Scaling Metrics for AI Systems** - **GPU Utilization**: Scale when GPUs are highly utilized across existing instances. - **Request Queue Depth**: Scale when pending requests exceed a threshold — indicates the current fleet can't keep up. - **Inference Latency**: Scale when the p95 or p99 latency exceeds SLA targets. - **Tokens Per Second**: Scale based on token throughput demand. - **Concurrent Requests**: Scale based on the number of simultaneous active requests. **Auto-Scaling Challenges for LLMs** - **Cold Start**: Loading a large model onto a new GPU takes **minutes** (model download, weight loading, CUDA initialization). This makes rapid scaling difficult. - **GPU Availability**: Cloud GPU instances are often scarce — scaling may fail if instances aren't available. - **Cost Spikes**: Auto-scaling during unexpected demand surges can cause dramatic cost increases. - **Minimum Scale**: Large models may require a minimum number of GPUs even at zero traffic, creating a high cost floor. **Solutions** - **Warm Pools**: Keep standby instances with models pre-loaded, ready to serve immediately. - **Scheduled Scaling**: Pre-scale for known traffic patterns (business hours, marketing campaigns). - **Spot/Preemptible Instances**: Use cheaper interruptible instances for burst capacity. - **Serverless Inference**: Services like **AWS SageMaker**, **Replicate**, and **Modal** handle scaling automatically. Auto-scaling is **essential** for cost-effective production AI — GPU compute is expensive, and paying for idle GPUs during off-peak hours is a significant waste.