atomic operations gpu cpu

**Atomic Operations in Parallel Computing** are **hardware-supported indivisible read-modify-write operations that guarantee correctness when multiple threads concurrently access shared memory locations — providing the foundation for lock-free data structures, parallel reductions, and thread-safe counters without the overhead of traditional mutex locks**. **Fundamental Atomic Operations:** - **Compare-and-Swap (CAS)**: atomically compares memory value to expected value and swaps with new value only if match — returns old value for caller to detect success/failure; foundation for nearly all lock-free algorithms - **Atomic Add/Sub**: atomically increments/decrements a memory location — used for counters, histogram building, and parallel reductions; hardware-accelerated on both CPUs (lock prefix) and GPUs (atomicAdd) - **Atomic Exchange**: atomically swaps a value into memory and returns the old value — useful for flag setting and simple lock acquisition - **Atomic Min/Max**: atomically updates memory with the minimum/maximum of current and new value — useful for parallel reduction to find extrema without explicit synchronization **CPU Atomic Semantics:** - **x86 LOCK Prefix**: cache line locked during atomic operation — prevents other cores from accessing the same line; costs 10-100 cycles depending on cache state (local: ~10 cycles, remote: ~100 cycles) - **Memory Ordering**: atomic operations serve as memory fences — acquire semantics prevent reordering of subsequent loads; release semantics prevent reordering of preceding stores; sequentially consistent (default in C++) provides both - **LL/SC (ARM)**: Load-Link/Store-Conditional pair — LL loads value, SC stores new value only if no other write occurred since LL; failure triggers retry loop; more flexible than CAS for complex atomic updates - **ABA Problem**: CAS succeeds incorrectly when value changes A→B→A between load and CAS — solved with version counters, tagged pointers, or hazard pointers in lock-free data structures **GPU Atomics:** - **Global Memory Atomics**: atomicAdd, atomicMax, atomicCAS on global memory — serialization at the L2 cache controller; throughput limited to ~1 atomic per 10 cycles per memory partition - **Shared Memory Atomics**: much faster (1-4 cycles) due to SM-local execution — used for per-block histograms and reductions before global aggregation - **Warp-Level Reduction Alternative**: __reduce_add_sync and warp shuffle can replace atomics for intra-warp operations — reduces atomic pressure by 32× by aggregating per-warp before one atomic per warp - **Atomic Contention Mitigation**: distribute atomic targets across multiple memory locations (privatization), then reduce — e.g., per-block histogram in shared memory, then atomicAdd to global histogram **Atomic operations are the essential synchronization primitive for high-performance parallel programming — mastering their use and understanding their performance characteristics enables developers to build scalable concurrent algorithms that avoid the serialization bottleneck of mutex-based synchronization.**