redundancy
**Redundancy** is the **reliability engineering principle of duplicating critical system components to eliminate single points of failure** — ensuring that production systems maintain availability and performance when individual servers, network links, storage volumes, or entire data centers fail, because in distributed computing the question is never whether components will fail but when, and redundancy determines whether failures are invisible to users or catastrophic.
**What Is Redundancy?**
- **Definition**: The intentional duplication of system components (hardware, software, data, network paths) so that the failure of any single component does not cause system unavailability.
- **Core Principle**: No single point of failure — every critical path has at least one backup that can assume the workload seamlessly.
- **Cost Trade-off**: Redundancy multiplies infrastructure cost but the cost of downtime (lost revenue, damaged reputation, SLA penalties) almost always exceeds the cost of redundancy.
- **ML Systems**: Model replicas, data store replication, distributed training checkpoints, and multi-region deployment are all forms of ML-specific redundancy.
**Types of Redundancy**
- **Active-Active**: All replicas serve production traffic simultaneously with load balancing distributing requests across them — maximum utilization and instant failover.
- **Active-Passive**: Standby components remain idle until the primary fails, then automatically activate — lower cost but brief failover delay.
- **N+1 Redundancy**: One extra component beyond the minimum required — balances cost efficiency with failure tolerance.
- **Geographic Redundancy**: Components distributed across multiple regions or availability zones for disaster recovery and latency optimization.
**Why Redundancy Matters**
- **Availability Guarantees**: Moving from 99% to 99.99% availability requires eliminating every single point of failure through redundancy.
- **Data Durability**: Data replication across multiple storage nodes protects against disk failures, corruption, and data loss.
- **Performance Under Failure**: With active-active redundancy, component failures reduce capacity but never eliminate service.
- **Disaster Recovery**: Geographic redundancy enables business continuity when entire data centers experience outages.
- **Compliance Requirements**: Financial, healthcare, and government regulations mandate minimum redundancy levels for critical systems.
**Redundancy in ML Systems**
| Component | Redundancy Strategy | Benefit |
|-----------|---------------------|---------|
| **Model Servers** | Multiple replicas behind load balancer | Inference survives pod failures |
| **Feature Store** | Replicated database with read replicas | Feature retrieval always available |
| **Training Checkpoints** | Stored across multiple storage backends | Training resumes after any failure |
| **Data Pipeline** | Idempotent stages with retry and replay | No data loss from transient failures |
| **Model Registry** | Replicated artifact storage | Models always deployable |
| **Monitoring** | Redundant alerting channels | Failures are always detected |
**Availability vs Redundancy**
| Availability Target | Annual Downtime | Typical Redundancy |
|---------------------|-----------------|---------------------|
| **99%** | 3.65 days | Basic redundancy |
| **99.9%** | 8.76 hours | N+1 with automated failover |
| **99.99%** | 52.6 minutes | Active-active, multi-AZ |
| **99.999%** | 5.26 minutes | Multi-region, active-active |
**Implementation Considerations**
- **Consistency Challenges**: Redundant data stores must handle replication lag and conflict resolution — CAP theorem constraints apply.
- **Cost Management**: Redundancy multiplies compute and storage costs — choose redundancy levels aligned with actual SLA requirements.
- **Failover Testing**: Redundancy only works if failover is tested regularly — untested failover paths fail when needed most.
- **Health Monitoring**: Redundancy requires robust health checks and automated failover triggers to work without human intervention.
- **Chaos Engineering**: Deliberately killing components in production validates that redundancy provides the expected protection.
Redundancy is **the foundational principle of reliable distributed systems** — transforming inevitable hardware and software failures from catastrophic outages into seamless, invisible events that users never notice, because production systems that matter must be designed to survive any single component failure without degradation.