Atomic Operations in Parallel Computing are hardware-supported indivisible read-modify-write operations that guarantee correctness when multiple threads concurrently access shared memory locations — providing the foundation for lock-free data structures, parallel reductions, and thread-safe counters without the overhead of traditional mutex locks.

Fundamental Atomic Operations:

• Compare-and-Swap (CAS): atomically compares memory value to expected value and swaps with new value only if match — returns old value for caller to detect success/failure; foundation for nearly all lock-free algorithms
• Atomic Add/Sub: atomically increments/decrements a memory location — used for counters, histogram building, and parallel reductions; hardware-accelerated on both CPUs (lock prefix) and GPUs (atomicAdd)
• Atomic Exchange: atomically swaps a value into memory and returns the old value — useful for flag setting and simple lock acquisition
• Atomic Min/Max: atomically updates memory with the minimum/maximum of current and new value — useful for parallel reduction to find extrema without explicit synchronization

CPU Atomic Semantics:

• x86 LOCK Prefix: cache line locked during atomic operation — prevents other cores from accessing the same line; costs 10-100 cycles depending on cache state (local: ~10 cycles, remote: ~100 cycles)
• Memory Ordering: atomic operations serve as memory fences — acquire semantics prevent reordering of subsequent loads; release semantics prevent reordering of preceding stores; sequentially consistent (default in C++) provides both
• LL/SC (ARM): Load-Link/Store-Conditional pair — LL loads value, SC stores new value only if no other write occurred since LL; failure triggers retry loop; more flexible than CAS for complex atomic updates
• ABA Problem: CAS succeeds incorrectly when value changes A→B→A between load and CAS — solved with version counters, tagged pointers, or hazard pointers in lock-free data structures

GPU Atomics:

• Global Memory Atomics: atomicAdd, atomicMax, atomicCAS on global memory — serialization at the L2 cache controller; throughput limited to ~1 atomic per 10 cycles per memory partition
• Shared Memory Atomics: much faster (1-4 cycles) due to SM-local execution — used for per-block histograms and reductions before global aggregation
• Warp-Level Reduction Alternative: __reduce_add_sync and warp shuffle can replace atomics for intra-warp operations — reduces atomic pressure by 32× by aggregating per-warp before one atomic per warp
• Atomic Contention Mitigation: distribute atomic targets across multiple memory locations (privatization), then reduce — e.g., per-block histogram in shared memory, then atomicAdd to global histogram

Atomic operations are the essential synchronization primitive for high-performance parallel programming — mastering their use and understanding their performance characteristics enables developers to build scalable concurrent algorithms that avoid the serialization bottleneck of mutex-based synchronization.