MapReduce

**MapReduce Programming Model** is **a distributed computing paradigm for processing massive datasets by mapping input to intermediate key-value pairs, shuffling by key, and reducing per-key values to final results** — enabling scalable batch processing on commodity clusters without explicit synchronization. MapReduce abstracts complexity of distributed computation. **Map Phase and Mappers** partition input data among mappers, each mapper applies user-defined function to input records, producing zero or more intermediate key-value pairs. Mappers run independently and in parallel—no communication required. Input typically comes from distributed file system with locality awareness: mappers run on nodes storing input data, reducing network traffic. **Shuffle and Sort Phase** automatically groups intermediate values by key, sorting keys for locality. System transfers output of all mappers to reducers handling their keys. Reducer receives all values for single key sorted, enabling single-pass processing. **Reduce Phase and Reducers** for each key, reducer applies user-defined function combining all values, producing final output. Reducer semantics: function should be associative and commutative to enable parallel operation. Many reducers run in parallel on different keys. **Combiner Optimization** applies reduce function locally on mapper output, reducing intermediate data size before shuffle. Particularly effective when reduce function is associative. **Partitioning and Locality** custom partitioner determines which reducer receives each key. Default hash partitioner distributes keys evenly. Locality-aware partitioning reduces network traffic. **Fault Tolerance** task failure detected by heartbeat mechanism. Failed mapper tasks re-executed from scratch, lost intermediate data reconstructed. Failed reducer tasks re-executed, reading intermediate data from persistent mapper output. **Stragglers and Speculative Execution** slow tasks (stragglers) delay job completion. Speculative execution runs backup copies of slow tasks, first copy to finish is used. Particularly effective for heterogeneous clusters. **Iterative Algorithms** MapReduce suits problems expressible as single map-reduce pairs. Iterative algorithms (e.g., k-means, PageRank) require multiple jobs. Each iteration's output becomes next iteration's input. **Skewed Datasets** with few hot keys become bottleneck—single reducer processes majority of data. Solutions include pre-grouping (multiple reducers per hot key) or custom skew-aware partitioning. **Applications** include word count, inverted index, data sort, distributed grep, log analysis. **MapReduce enables simple expression of distributed algorithms** without explicit synchronization, network programming, or failure handling.