mapreduce hadoop distributed

**MapReduce and Hadoop Ecosystem: Disk-Based Distributed Computing — foundational framework for batch processing at scale** MapReduce is a programming model for distributed batch processing: map phase (process input key-value pairs, emit intermediate pairs), shuffle and sort (group intermediate pairs by key), reduce phase (aggregate values per key). Hadoop implements MapReduce over HDFS, enabling massive data-parallel computations on commodity clusters. **HDFS Architecture** Hadoop Distributed File System (HDFS) replicates data blocks (default 3x) across nodes for fault tolerance and locality-aware task scheduling. Namenode manages namespace and file system tree; datanodes store blocks and perform low-level read/write operations. Block size (default 128 MB, configurable to 256 MB or larger) determines parallelism: one map task per block enables fine-grained locality. Read operations retrieve from nearest replica; write operations use pipelined striping across replicas. **MapReduce Job Execution** Mapper instances (one per HDFS block) read data, apply user function, emit intermediate key-value pairs. Hadoop sorts and partitions intermediate data by key, distributing partitions to reducers. Shuffle phase (network-intensive) transfers intermediate data from mappers to reducers. Reducer instances (user-configurable count) aggregate values per key, outputting final results. Speculative execution re-runs slow tasks on backup nodes, improving tail latency. **YARN Resource Manager** YARN (Yet Another Resource Negotiator) separates cluster resource management from computation. Resource Manager (global) maintains cluster state; Node Managers report per-node resources and container lifecycle. Applications request containers (CPU cores, memory); RM allocates containers via scheduling policies (FIFO, Fair, Capacity). MapReduce and other frameworks (Spark, HBase) run atop YARN as clients. **Ecosystem and Decline** Hive provides SQL interface atop MapReduce, translating queries to MapReduce jobs. HBase adds random-access capabilities via LSM trees. Pig enables dataflow scripting with automatic MapReduce compilation. Combiners reduce intermediate data volume pre-shuffle. However, Spark's in-memory caching and DAG scheduling outperformed Hadoop MapReduce by 10-100x on iterative workloads, causing Hadoop's decline in modern data pipelines.