MapReduce and Hadoop Ecosystem: Disk-Based Distributed Computing — foundational framework for batch processing at scale

MapReduce is a programming model for distributed batch processing: map phase (process input key-value pairs, emit intermediate pairs), shuffle and sort (group intermediate pairs by key), reduce phase (aggregate values per key). Hadoop implements MapReduce over HDFS, enabling massive data-parallel computations on commodity clusters.

HDFS Architecture

Hadoop Distributed File System (HDFS) replicates data blocks (default 3x) across nodes for fault tolerance and locality-aware task scheduling. Namenode manages namespace and file system tree; datanodes store blocks and perform low-level read/write operations. Block size (default 128 MB, configurable to 256 MB or larger) determines parallelism: one map task per block enables fine-grained locality. Read operations retrieve from nearest replica; write operations use pipelined striping across replicas.

MapReduce Job Execution

Mapper instances (one per HDFS block) read data, apply user function, emit intermediate key-value pairs. Hadoop sorts and partitions intermediate data by key, distributing partitions to reducers. Shuffle phase (network-intensive) transfers intermediate data from mappers to reducers. Reducer instances (user-configurable count) aggregate values per key, outputting final results. Speculative execution re-runs slow tasks on backup nodes, improving tail latency.

YARN Resource Manager

YARN (Yet Another Resource Negotiator) separates cluster resource management from computation. Resource Manager (global) maintains cluster state; Node Managers report per-node resources and container lifecycle. Applications request containers (CPU cores, memory); RM allocates containers via scheduling policies (FIFO, Fair, Capacity). MapReduce and other frameworks (Spark, HBase) run atop YARN as clients.

Ecosystem and Decline

Hive provides SQL interface atop MapReduce, translating queries to MapReduce jobs. HBase adds random-access capabilities via LSM trees. Pig enables dataflow scripting with automatic MapReduce compilation. Combiners reduce intermediate data volume pre-shuffle. However, Spark's in-memory caching and DAG scheduling outperformed Hadoop MapReduce by 10-100x on iterative workloads, causing Hadoop's decline in modern data pipelines.