Load balancing distributes work across healthy service instances to meet throughput, latency, locality and availability goals. AI inference needs routing that understands model identity, accelerator memory, KV-cache affinity, batching opportunity and heterogeneous capacity, not merely server count. Layer 4 balancers route connections using transport metadata; Layer 7 balancers inspect HTTP/gRPC requests, identity, model and headers and can apply richer policy at added cost. A production definition states the service or pipeline boundary, tenants, workload and data classes, dependency graph, consistency and durability expectations, capacity envelope, latency and availability objectives, failure model, trust zones, deployment units, ownership, and evidence required for release. Architecture diagrams and service-level indicators must refer to the same boundary. Define request unit, health, session affinity, weights, locality, retry ownership, timeout, overload behavior, consistency, fail-open/closed policy and success metrics.

Architecture, control plane, and operating behavior. Clients reach a global or regional front door, an L4/L7 tier selects a pool, a scheduler chooses replicas, health probes remove failures, and observability updates weights. Stateful caches and streaming sessions may require consistent hashing or explicit ownership. Round robin cycles evenly, weighted round robin reflects capacity, least connections approximates outstanding work, least latency uses feedback, power-of-two choices reduces coordination, and consistent hashing limits remapping for stateful keys. DNS/global, anycast, hardware appliance, software proxy, service mesh, Kubernetes service, client-side, queue-based and model-aware GPU schedulers operate at different boundaries. The operational stack spans clients and producers, APIs or ingestion, queues and schedulers, stateless and stateful compute, accelerators, memory and storage, network fabrics, identity and policy, artifact registries, observability, automation, and human operations. Control-plane decisions and data-plane work are separated so overload or compromise in one does not silently corrupt the other. Evaluation combines correctness and model quality with throughput, p50/p95/p99 latency, queue depth, saturation, availability, error and retry rates, freshness, data loss, recovery time, recovery point, capacity, utilization, memory, network, energy, cost, and operator toil. Service-level objectives use user-visible good events, explicit windows, and error budgets rather than infrastructure uptime alone.

Implementation, infrastructure, and failure modes. Separate liveness from readiness, use outlier detection, slow start, bounded retries, connection draining, locality preferences, overload admission, circuit breaking and load-shedding. Avoid synchronized probes and make weights explainable. NIC, CPU proxy, TLS, network hops, accelerator HBM, model residency, KV cache, batch queues and interconnect shape capacity. GPU utilization alone misses memory and decode-stage pressure. Retry amplification, sticky hot keys, stale health, flapping endpoints, uneven long requests, cross-zone traffic, connection imbalance, queue starvation and thundering herds can make redundancy worsen outages. Implementation favors immutable artifacts, declarative configuration, typed schemas, idempotent operations, bounded retries with jitter, deadlines, backpressure, health and readiness probes, least privilege, encrypted transport and storage, progressive rollout, reproducible environments, and complete telemetry. Automation has dry-run, approval, audit, and rollback paths. AI infrastructure joins CPUs, GPUs or NPUs, HBM, host memory, NICs and DPUs, PCIe and scale-up links, leaf-spine networks, local and shared storage, power delivery, and cooling. Topology, NUMA locality, bandwidth, failure domains, thermal headroom, and accelerator memory determine delivered behavior and must be visible to schedulers. Common failures include retry storms, queue collapse, stale health signals, split brain, partial writes, incompatible schemas, silent data corruption, time skew, dependency amplification, capacity fragmentation, noisy neighbors, credential leakage, unbounded state, monitoring blind spots, and recovery procedures that exist only on paper. A healthy component does not prove a healthy user journey.

Verification, security, and lifecycle controls. Use skewed sizes and keys, long streams, unhealthy/slow instances, zone loss, cold replicas, cache affinity, overload, connection churn, retry faults and latency-throughput sweeps. Request goodput, p99 latency, queue time, imbalance, utilization, error, retry, ejection, cache hit, cross-zone bytes, batch efficiency, dropped work and cost matter. Routing must enforce tenant, residency, model entitlement, rate limits, isolation, audit and safe failure. Health endpoints reveal minimal information. Verification combines unit, contract and property tests, schema compatibility, load and soak tests, chaos and fault injection, security review, backup restoration, failover and rollback drills, dependency degradation, regional evacuation where applicable, data reconciliation, shadow traffic, canaries, and end-to-end synthetic checks. Tests run against production-like scale and permissions. Source, data, configuration, environment, model, registry metadata, infrastructure definition, dependency, image, driver, firmware, deployment, experiment, approval, incident, and rollback artifacts remain linked. Continuous controls detect drift, expired credentials, unowned resources, stale backups, regressions, policy exceptions, and unsupported versions. Owners define access, segregation of duties, data classification, residency, retention and deletion, vendor and supply-chain review, incident severity, communications, audit evidence, RTO/RPO or SLO exceptions, cost attribution, and change authority. Sensitive model and experiment artifacts receive the same integrity and confidentiality controls as source and production data.

<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif"><rect width="760" height="470" fill="#0d1117"/><defs><marker id="arrow" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="6" markerHeight="6" orient="auto"><path d="M0 0L10 5L0 10Z" fill="#60a5fa"/></marker><marker id="green" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="6" markerHeight="6" orient="auto"><path d="M0 0L10 5L0 10Z" fill="#34d399"/></marker></defs><text x="380" y="34" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">Load Balancing — Distribute Requests Without Hotspots</text><text x="380" y="56" fill="#8b98a5" font-size="13" text-anchor="middle">a health-aware scheduler routes work across unequal backends and removes failed instances</text><g transform="translate(36 177)"><g fill="#10233b" stroke="#60a5fa"><circle cx="28" cy="28" r="25"/><circle cx="28" cy="91" r="25"/><circle cx="28" cy="154" r="25"/></g><g fill="#93c5fd"><circle cx="22" cy="23" r="4"/><circle cx="35" cy="23" r="4"/><circle cx="22" cy="86" r="4"/><circle cx="35" cy="86" r="4"/><circle cx="22" cy="149" r="4"/><circle cx="35" cy="149" r="4"/></g><text x="28" y="196" fill="#8b98a5" font-size="9.5" text-anchor="middle">clients</text></g><path d="M91 268H172" stroke="#60a5fa" stroke-width="3" marker-end="url(#arrow)"/><g transform="translate(180 190)"><path d="M0 78L78 0l78 78-78 78Z" fill="#211936" stroke="#a78bfa" stroke-width="2"/><text x="78" y="68" fill="#c4b5fd" font-size="11" font-weight="700" text-anchor="middle">load balancer</text><text x="78" y="87" fill="#8b98a5" font-size="9.5" text-anchor="middle">health + load + locality</text></g><path d="M338 242C388 242 377 124 426 124M338 268H426M338 294C388 294 377 386 426 386" fill="none" stroke="#a78bfa" stroke-width="2.5" marker-end="url(#arrow)"/><g transform="translate(434 91)"><rect width="105" height="64" rx="7" fill="#101a28" stroke="#34d399" stroke-width="2"/><g fill="#34d399"><circle cx="16" cy="17" r="4"/><circle cx="16" cy="32" r="4"/><circle cx="16" cy="47" r="4"/></g><path d="M29 17h59M29 32h46M29 47h54" stroke="#64748b"/><text x="52" y="81" fill="#34d399" font-size="9.5" text-anchor="middle">backend A · 42%</text></g><g transform="translate(434 236)"><rect width="105" height="64" rx="7" fill="#101a28" stroke="#34d399" stroke-width="2"/><g fill="#34d399"><circle cx="16" cy="17" r="4"/><circle cx="16" cy="32" r="4"/><circle cx="16" cy="47" r="4"/></g><path d="M29 17h59M29 32h46M29 47h54" stroke="#64748b"/><text x="52" y="81" fill="#34d399" font-size="9.5" text-anchor="middle">backend B · 37%</text></g><g transform="translate(434 353)"><rect width="105" height="64" rx="7" fill="#101a28" stroke="#f87171" stroke-width="2"/><g fill="#f87171"><circle cx="16" cy="17" r="4"/><circle cx="16" cy="32" r="4"/><circle cx="16" cy="47" r="4"/></g><path d="M29 17h59M29 32h46M29 47h54" stroke="#64748b"/><text x="52" y="81" fill="#f87171" font-size="9.5" text-anchor="middle">backend C · unhealthy</text></g><g transform="translate(584 88)"><path d="M0 300V0M0 300H131" stroke="#3a4453"/><g fill="#60a5fa"><rect x="15" y="121" width="22" height="179"/><rect x="54" y="142" width="22" height="158"/><rect x="93" y="272" width="22" height="28"/></g><line x1="0" y1="83" x2="131" y2="83" stroke="#f59e0b" stroke-dasharray="5 4"/><text x="126" y="76" fill="#fbbf24" font-size="9" text-anchor="end">capacity</text><g fill="#8b98a5" font-size="9" text-anchor="middle"><text x="26" y="320">A</text><text x="65" y="320">B</text><text x="104" y="320">C</text></g><text x="65" y="20" fill="#8b98a5" font-size="9.5" text-anchor="middle">in-flight requests</text></g><path d="M486 352C476 313 407 315 321 328" fill="none" stroke="#f87171" stroke-width="2" stroke-dasharray="6 4" marker-end="url(#arrow)"/><text x="402" y="347" fill="#fca5a5" font-size="9.5" text-anchor="middle">failed health check → drain</text><path d="M486 171C545 184 554 206 584 230" fill="none" stroke="#34d399" stroke-width="2" marker-end="url(#green)"/><text x="380" y="452" fill="#6b7684" font-size="11.5" text-anchor="middle">Good balancing minimizes tail latency while respecting capacity, session affinity, failure domains, and connection-drain semantics.</text></svg>

Selection and production application. Use round robin for homogeneous stateless pools, least-work policy for variable requests, consistent hashing for state affinity, and model-aware queueing for GPU inference. Web APIs, model serving, microservices, storage, databases, streaming, batch schedulers and distributed compute rely on load balancing. Load balancing interacts with autoscaling, caching, model placement, health, retries, service discovery, network, admission control and SLOs. The useful optimization and reliability boundary is the complete user-facing system. Improving a model server, network, registry, deployment controller, or pipeline stage can move the bottleneck or weaken consistency, safety, recoverability, and cost elsewhere, so decisions are validated end to end. A production definition states the service or pipeline boundary, tenants, workload and data classes, dependency graph, consistency and durability expectations, capacity envelope, latency and availability objectives, failure model, trust zones, deployment units, ownership, and evidence required for release. Architecture diagrams and service-level indicators must refer to the same boundary. Evaluation combines correctness and model quality with throughput, p50/p95/p99 latency, queue depth, saturation, availability, error and retry rates, freshness, data loss, recovery time, recovery point, capacity, utilization, memory, network, energy, cost, and operator toil. Service-level objectives use user-visible good events, explicit windows, and error budgets rather than infrastructure uptime alone. CFS connects this topic to semiconductor architecture, implementation, verification, manufacturing, packaging, test, and deployed AI-system tradeoffs across the platform.