distributed computing framework
**Distributed Computing Frameworks** are the **software platforms that abstract the complexity of executing parallel computations across clusters of networked machines — handling task distribution, data partitioning, fault tolerance, and result aggregation so that programmers can express parallel algorithms without managing the underlying distributed systems plumbing**.
**The Distributed Computing Challenge**
Moving from a single machine to a cluster introduces fundamental challenges absent in shared-memory parallelism: network latency (~1-100 us), partial failures (any node can crash independently), data locality (moving computation to data is cheaper than moving data to computation), and heterogeneous performance (straggler mitigation).
**Key Frameworks**
- **MapReduce (Hadoop)**: The foundational distributed computing model. Map phase: user-defined function processes each input record independently, emitting key-value pairs. Shuffle: the framework redistributes pairs by key across nodes. Reduce phase: user-defined function aggregates all values for each key. Fault tolerance through deterministic re-execution. Hadoop's disk-based shuffle made it slow for iterative workloads.
- **Apache Spark**: In-memory distributed computing that overcomes MapReduce's disk I/O bottleneck. Resilient Distributed Datasets (RDDs) cache intermediate results in memory across iterations — 10-100x faster than Hadoop for iterative/interactive workloads. Spark's DAG scheduler optimizes multi-stage pipelines and handles lineage-based fault recovery (recompute lost partitions from recorded transformations).
- **Dask**: Python-native distributed computing. Extends NumPy/Pandas APIs to out-of-core and distributed datasets. Dask DataFrames partition a logical DataFrame across workers; Dask Arrays partition an N-dimensional array. The task graph scheduler dynamically executes operations across a local or distributed cluster with minimal code change from single-machine Pandas/NumPy.
- **Ray**: General-purpose distributed execution framework. @ray.remote decorator converts any Python function into a distributed task and any class into a distributed actor. Task-level parallelism with dynamic scheduling and object store (shared memory + distributed references) for efficient data sharing. Powers distributed training (Ray Train), hyperparameter tuning (Ray Tune), and reinforcement learning (RLlib).
**Data Partitioning Strategies**
- **Hash Partitioning**: Assign records to partitions by hash(key) mod P. Ensures even distribution for joins and aggregations by key.
- **Range Partitioning**: Assign records to partitions by key range. Preserves sort order — useful for range queries.
- **Locality-Aware Partitioning**: Co-locate partitions that are frequently joined, reducing shuffle traffic.
**Fault Tolerance Mechanisms**
- **Lineage-Based Recovery (Spark)**: If a partition is lost, recompute it from the original data using the recorded transformation chain. No replication overhead.
- **Checkpointing**: Periodically save intermediate state to durable storage. Truncates the lineage chain for long computations.
- **Speculative Execution**: Duplicate slow tasks on other nodes. Use whichever copy finishes first (straggler mitigation).
Distributed Computing Frameworks are **the operating systems of cluster-scale parallel computation** — providing the abstractions that let data scientists and engineers think about algorithms and data transformations rather than network protocols, failure handling, and task scheduling.