communication avoiding algorithms

**Communication-Avoiding Algorithms** — Algorithm designs that minimize data movement between levels of the memory hierarchy and between processors, achieving provably optimal communication bounds. **Communication Lower Bounds** — The Hong-Kung red-blue pebble game establishes lower bounds on data movement for computational directed acyclic graphs. For matrix multiplication with fast memory of size M, the minimum number of words transferred is Omega(N^3 / sqrt(M)), regardless of the algorithm's schedule. These bounds apply to both sequential cache transfers and parallel inter-processor communication. Matching these lower bounds requires fundamentally restructuring algorithms rather than simply tuning existing implementations. **2.5D and 3D Matrix Multiplication** — Classical 2D parallel matrix multiplication distributes an NxN matrix across P processors in a sqrt(P) x sqrt(P) grid, requiring O(N^2 / sqrt(P)) communication per processor. The 2.5D algorithm replicates data across c copies, reducing communication by a factor of sqrt(c) at the cost of c times more memory. When c = P^(1/3), the 3D algorithm achieves the optimal communication bound of O(N^2 / P^(2/3)). This tradeoff between memory and communication generalizes to many linear algebra operations. **Communication-Avoiding Krylov Methods** — Standard Krylov solvers like GMRES and CG perform one sparse matrix-vector multiplication and one or two global reductions per iteration. Communication-avoiding variants compute s iterations worth of Krylov basis vectors using a single matrix powers kernel, reducing synchronization points by a factor of s. The s-step Lanczos and s-step CG algorithms require careful numerical stabilization through techniques like Newton or Chebyshev basis polynomials to maintain orthogonality. These methods can achieve 2-10x speedups on large-scale distributed systems where global synchronization is expensive. **Cache-Oblivious Parallel Approaches** — Cache-oblivious algorithms achieve optimal cache performance without knowing cache parameters by using recursive divide-and-conquer decomposition. Parallel cache-oblivious matrix multiplication recursively splits matrices into quadrants, naturally exploiting locality at every cache level. The Cilk runtime's work-stealing scheduler preserves cache-oblivious locality guarantees when executing these recursive algorithms in parallel. Tall-cache assumptions where M >= B^2 for cache size M and line size B are sufficient for most cache-oblivious algorithms to achieve optimal bounds. **Communication-avoiding algorithms represent a paradigm shift in algorithm design, achieving asymptotically fewer data transfers and enabling parallel applications to scale efficiently on modern memory hierarchies and distributed systems.**