Apache Spark: In-Memory DAG Execution — enabling 10-100x faster iterative analytics versus Hadoop MapReduce

Apache Spark is a distributed computing framework centered on RDDs (Resilient Distributed Datasets) and lazy evaluation. RDDs represent immutable distributed collections with lineage DAGs (directed acyclic graphs) enabling fault tolerance via recomputation.

RDD and Lineage DAG

RDDs are partitioned across cluster nodes, enabling parallel operations. Creation via transformation (map, filter, join) produces new RDDs linked to parents, forming lineage DAGs. On action (collect, save, count), Spark traverses DAG backward, identifies missing partitions, schedules stage (set of tasks with no shuffle), and executes via task scheduler. Lineage enables recovery: if partition N is lost, Spark recomputes from upstream. This lazy evaluation enables optimization: Spark analyzes full DAG before execution, fusing operations (map-map fusion), eliminating redundant shuffles.

Catalyst Optimizer

Spark SQL queries transform into optimized plans via Catalyst: logical plan (operators representing computation), physical plan (execution strategies per operator), and code generation. Predicate pushdown eliminates unnecessary data early; join reordering minimizes intermediate data volume. Generated code uses (just-in-time) compilation via Janino, achieving near-hand-written performance.

DataFrames and Dataset API

DataFrames provide SQL interface (relational tables), abstracting RDD complexity. Datasets (Scala/Java) offer type safety while retaining performance. Both leverage Catalyst optimization, significantly outperforming raw RDD operations on SQL-like workloads.

In-Memory Caching and Spill

RDD.cache() persists partitions in memory, enabling sub-second reuse versus 10-100ms disk latency. Least-recently-used (LRU) eviction spills excess partitions to disk when memory pressure exceeds thresholds. Iterative machine learning algorithms (gradient descent) cache data, achieving 10-100x speedup over disk-based MapReduce.

Spark Streaming and Structured Streaming

Spark Streaming ingests data in micro-batches (50-500 ms intervals), enabling second-scale latency. Structured Streaming (Spark 2.0+) provides continuous execution model and event-time semantics via watermarking. Both leverage Spark's optimization and fault tolerance.