OpenMP Parallel Programming provides a pragmatic, standards-based API for shared-memory parallelism using directives, enabling rapid parallel code development without explicit thread management.

Fork-Join Model and Pragma Syntax

• OpenMP Execution Model: Main thread creates team of worker threads at parallel regions. Workers execute concurrently, rejoin at implicit barrier.
• Pragma Syntax: #pragma omp parallel directives inserted before loops/code blocks. Preprocessor expands pragmas; implicit compiler code generation.
• Region Definition: #pragma omp parallel creates team. Implicit barrier at end (threads wait for all to complete before proceeding).
• Multiple Region Types: parallel, parallel for, parallel sections, parallel critical. Each combines task distribution with synchronization.

Parallel For Loops and Work Distribution

• #pragma omp parallel for: Divides loop iterations across threads. Implicit team creation + loop distribution + implicit barrier.
• Static Scheduling: Iterations 0-N divided into chunks allocated at compile time. Thread i gets chunk i. Good for balanced loops, poor for variable iteration counts.
• Dynamic Scheduling: Chunks grabbed by threads as they finish previous chunks. Good for imbalanced loops (iterations vary in time), higher overhead.
• Guided Scheduling: Chunk size decreases as loop progresses. Reduces overhead vs full dynamic while maintaining load balance.

Reduction and Shared/Private Variable Clauses

• Reduction Clause: #pragma omp parallel for reduction(+:sum) accumulates partial sums from threads into global sum. Prevents race conditions.
• Supported Operators: +, -, *, /, &, |, ^, &&, || for integer; min, max. Custom reductions via user-defined operations.
• Shared Clause: Variables marked shared accessible to all threads (synchronization required). Implicit for global variables.
• Private Clause: Each thread gets independent copy initialized at region entry. Implicit for loop counters, scalars.
• Critical Section: #pragma omp critical serializes updates (only one thread enters at a time). Lower overhead than mutex but serialized.

Task Parallelism (OpenMP 4.0+)

• omp task Directive: Generates task for asynchronous execution. Parent thread enqueues task; worker threads execute when available.
• Recursive Parallelism: Quicksort, tree traversal naturally expressed via tasks. Each task spawns subtasks, creating dynamic task tree.
• Task Dependencies: #pragma omp task depend(in:A) depends(out:B) specifies data dependencies. Runtime scheduler respects dependencies, enabling asynchronous execution.
• Taskgroup: #pragma omp taskgroup creates barrier for all spawned tasks. Ensures tasks complete before proceeding.

SIMD Vectorization Directives

• #pragma omp simd: Compiler unrolls loop for vectorization (SIMD units: AVX-512, NEON, etc.). Compiler generates vector instructions for supported data types.
• Vector Length Control: pragma omp simd simdlen(16) requests specific vector width. Compiler uses widest available that supports simdlen.
• Collapse: #pragma omp simd collapse(2) enables vectorization across nested loops. Collapses 2D loop into 1D for better vectorization.
• Reduction + SIMD: omp simd reduction(+:sum) combines loop unrolling with reduction. Compiler uses vector units for partial sums.

Nested Parallelism

• Nested Parallel Regions: Inner parallel regions create additional thread levels. Threads nested up to implementation limits (typically 2-3 levels).
• omp_get_num_levels(): Query nesting depth. omp_get_ancestor_thread_num() identify ancestor threads in hierarchy.
• Performance Considerations: Excessive nesting reduces SIMD width per thread (threads per core), increases synchronization overhead. Typically avoid >2 levels.

OpenMP 5.0 Target Offloading to GPU

• #pragma omp target: Offload computation to GPU. Similar to CUDA but uses OpenMP syntax.
• Target Data: #pragma omp target data map(to:A[0:N]) specifies data transfer (host to device). Avoided repeated transfers.
• Parallel Teams: #pragma omp target teams parallel for combines multiple levels of parallelism (multiple blocks of multiple threads).
• GPU Kernels: omp target regions compile to GPU kernels. NVIDIA/AMD/Intel compilers generate ISA-specific code.

Real-World Applications and Performance

• Adoption: OpenMP standard in scientific/HPC communities (Fortran, C/C++). ~80% of HPC codes use OpenMP for shared-memory parallelism.
• Performance Predictability: Static scheduling easier to profile/optimize; dynamic scheduling less predictable.
• Compiler Variability: Different compilers generate different code quality. Intel icc often outperforms GCC/Clang for OpenMP.
• Hybrid Paradigms: MPI (distributed memory) + OpenMP (shared-memory within node) dominant in HPC. Scales 100s-1000s cores across clusters.