fault tolerance

**Fault tolerance** is the ability of a system to **continue operating correctly** even when one or more of its components fail. Unlike high availability (which focuses on uptime percentage), fault tolerance emphasizes **correct behavior during failures** — the system doesn't just stay up, it produces correct results. **Fault Tolerance Principles** - **Redundancy**: Run multiple copies of critical components so failure of one doesn't affect the system. - **Isolation**: Contain failures so they don't cascade — a crashed RAG service shouldn't take down inference. - **Detection**: Quickly identify failed components through health checks, heartbeats, and monitoring. - **Recovery**: Automatically replace or restart failed components without human intervention. **Fault Tolerance Techniques** - **Replication**: Run multiple instances of services. Use consensus protocols (Raft, Paxos) for stateful services. - **Circuit Breaker**: Stop calling a failing service and route to alternatives. Prevents cascading failures. - **Bulkhead Pattern**: Isolate resources so one failing tenant, user, or request type can't exhaust resources for others. - **Retry with Fallback**: Retry failed operations, then fall back to alternatives if retries are exhausted. - **Timeout Enforcement**: Ensure no single operation can hang indefinitely and block resources. - **Graceful Degradation**: Provide reduced functionality rather than complete failure. Serve cached responses, simpler model outputs, or template responses when primary systems fail. **Fault Tolerance in AI Systems** - **Model Fallback Chain**: Primary model → backup model → cached response → template response → "I'm unable to answer right now." - **RAG Resilience**: If the vector database is down, the model can still answer from parametric knowledge (with reduced quality). - **Distributed Inference**: Frameworks like vLLM can handle individual GPU failures in a multi-GPU setup. - **Data Pipeline Resilience**: If a data source for RAG is unavailable, use the last cached version. **Testing Fault Tolerance** - **Chaos Engineering**: Deliberately inject failures (kill processes, drop network connections, fill disks) and verify the system continues operating correctly. - **Game Days**: Planned events where the team practices responding to simulated failures. - **Netflix's Chaos Monkey**: Randomly kills production instances to verify fault tolerance. Fault tolerance is the difference between "the system stayed up" and "the system stayed up **and gave correct answers**" during failures.