Apache

**Apache Spark Distributed Computing** is **a fast, distributed computing framework providing in-memory data processing with fault tolerance through Resilient Distributed Datasets (RDDs), enabling iterative algorithms and interactive analysis at scale** — successor to MapReduce with better performance for iterative workloads. Spark unifies batch, streaming, and interactive processing. **Resilient Distributed Datasets (RDD)** are immutable, distributed collections fault-tolerant through lineage: RDD knows its parent RDDs and transformation applied, enabling recomputation on failure. Lazy evaluation: transformations don't execute immediately, only when action triggered. Lazy semantics enable optimization. **Transformations and Actions** transformations (map, filter, flatMap, join, reduceByKey) create new RDDs from existing ones. Actions (collect, save, count) return results or write to storage, triggering execution. **Wide vs. Narrow Transformations** narrow transformations (map, filter) map each input partition to one output partition, no shuffle. Wide transformations (shuffle, sort, reduceByKey) multiple input partitions map to output partitions, requiring network shuffle. Understanding width guides performance optimization. **Caching and Persistence** frequently-accessed RDDs cached in memory (persist()), avoiding recomputation. Cache levels: MEMORY_ONLY (fast, may evict), MEMORY_AND_DISK (swap to disk), DISK_ONLY, or replication variants. **Partitioning and Locality** data partitioned across cluster nodes. Spark respects HDFS block locality: task runs on node storing data. **Spark SQL and DataFrames** optimized interface for structured data. DataFrames provide relational API (select, where, groupBy), Catalyst optimizer generates efficient execution plans. Much faster than low-level RDD operations. **Streaming and Micro-Batches** Spark Streaming discretizes continuous data into micro-batches, enabling RDD operations on stream. DStream = sequence of RDDs. **Catalyst Optimizer** analyzes logical execution plans, optimizes: predicate pushdown (filter near source), projection pruning (select only needed columns), join reordering. **Shuffle and Sort Bottlenecks** wide transformations trigger network shuffle—expensive. Minimizing shuffles improves performance. **Graph Processing (GraphX)** distributed graph processing API on top of RDDs. **Machine Learning Library (MLlib)** distributed ML algorithms: clustering, classification, regression, recommendation. **Applications** include ETL, data warehousing, streaming analytics, graph analytics, machine learning. **Spark's in-memory caching and lazy evaluation enable dramatic performance improvements over MapReduce** for iterative and interactive workloads.