vertical scaling

**Vertical scaling** (scaling up) is the practice of adding **more resources to an existing machine** — more CPU cores, more RAM, faster or more GPUs, faster storage — to handle increased workload. It is the simplest scaling approach but has physical upper limits. **How Vertical Scaling Works** - **Upgrade Hardware**: Add more GPU memory, upgrade to a faster GPU, increase RAM, add NVMe storage. - **Same Application**: The application code doesn't need to change — it simply has more resources available. - **No Distribution Complexity**: No need for load balancers, distributed state, or inter-node communication. **Vertical Scaling for AI/ML** - **Larger GPU**: Move from **A10G** (24GB) to **A100** (80GB) to **H100** (80GB) for more VRAM and faster inference. - **Multi-GPU Single Node**: Use **NVIDIA DGX** or cloud instances with 4–8 GPUs connected via **NVLink** for high-bandwidth inter-GPU communication. - **More RAM**: Increase system RAM to support larger batch sizes, bigger KV caches, or more concurrent requests. - **Faster Storage**: Use NVMe SSDs for faster model loading and checkpointing. **Advantages** - **Simplicity**: No distributed systems complexity — the application runs on one machine. - **Low Latency**: No network hops between components — all communication is in-memory. - **Easy Management**: One machine to monitor, maintain, and debug. - **Better for Large Models**: Models that require multi-GPU parallelism benefit from high-bandwidth intra-node GPU connections (NVLink) rather than inter-node networking. **Limitations** - **Hardware Ceiling**: Even the most powerful single machine has limits — currently an 8×H100 DGX with 640GB total GPU memory. - **Single Point of Failure**: If the machine goes down, the entire service is unavailable. - **Cost Scaling**: High-end hardware has **non-linear cost** — a 2× more powerful machine often costs 3–4× more. - **Downtime for Upgrades**: Hardware upgrades typically require system downtime. **When to Choose Vertical Scaling** - Early-stage development and prototyping. - Models that fit on a single (possibly multi-GPU) machine. - Latency-sensitive applications where network hops are unacceptable. Vertical scaling is often the **right first step** before investing in the complexity of horizontal scaling — maximize what a single machine can do before distributing.