horizontal scaling
**Horizontal scaling** (scaling out) is the practice of adding **more machines** to a system to handle increased load, distributing work across a growing fleet of servers. It is the primary scaling strategy for production AI systems because LLM inference is compute-intensive and single machines have physical limits.
**How Horizontal Scaling Works**
- **Add Instances**: Deploy additional server instances running the same service.
- **Load Balancer**: A load balancer distributes incoming requests across all instances.
- **Shared Nothing**: Each instance is independent — no shared memory or local state between instances.
- **Stateless Design**: Services must be designed to be stateless (or use external state stores) so any instance can handle any request.
**Horizontal vs. Vertical Scaling**
| Aspect | Horizontal (Scale Out) | Vertical (Scale Up) |
|--------|----------------------|--------------------|
| **Method** | Add more machines | Add more resources to existing machine |
| **Limit** | Practically unlimited | Physical hardware limits |
| **Downtime** | No downtime to add instances | Often requires restart |
| **Cost** | Many cheaper machines | Single expensive machine |
| **Complexity** | Higher (distributed systems) | Lower (single machine) |
**Horizontal Scaling for AI/ML**
- **Inference Scaling**: Deploy multiple GPU servers running the same model, distribute inference requests across them.
- **Data Parallelism**: Distribute training data across GPUs/machines, each computing gradients on a subset.
- **Pipeline Parallelism**: Split model layers across machines, processing different microbatches simultaneously.
- **RAG Scaling**: Distribute vector database shards across multiple nodes for higher query throughput.
**Challenges**
- **Model Size**: Large models (70B+ parameters) may not fit on a single GPU, requiring **model parallelism** before horizontal scaling applies.
- **Consistency**: Ensuring all instances serve the same model version during rolling deployments.
- **Cost Efficiency**: GPU machines are expensive, so right-sizing instance count is critical.
Horizontal scaling is the **industry standard** approach for handling production LLM traffic — all major AI API providers (OpenAI, Anthropic, Google) use large fleets of GPU servers behind load balancers.