parallel debugging

**Parallel Debugging and Correctness Tools** are the **specialized analysis and detection systems that identify concurrency bugs — race conditions, deadlocks, atomicity violations, and memory ordering errors — that are fundamentally harder to find than sequential bugs because they depend on non-deterministic thread interleavings that may occur once per million executions and are nearly impossible to reproduce reliably**. **Why Parallel Bugs Are Different** A sequential bug is deterministic — the same input produces the same failure every time. A concurrency bug depends on the relative timing of multiple threads, which is affected by CPU load, cache state, OS scheduling, and even temperature. A race condition might manifest as a crash on a customer's 128-core server but be completely unreproducible on the developer's 8-core workstation. **Categories of Concurrency Bugs** - **Data Race**: Two threads access the same memory location concurrently, at least one writes, and there is no synchronization (lock, atomic, barrier) ordering the accesses. The result depends on which thread executes first — undefined behavior in C/C++. - **Deadlock**: Two or more threads each hold a lock and wait for a lock held by another thread. The circular dependency means none can proceed. The program freezes. - **Atomicity Violation**: A sequence of operations that the programmer assumes is atomic (indivisible) is actually interruptible. Example: check-then-act (`if (ptr != NULL) use(ptr)`) where ptr is set to NULL by another thread between the check and the use. - **Order Violation**: Operations from different threads execute in an unexpected order. Often caused by missing memory barriers on relaxed architectures. **Detection Tools** - **ThreadSanitizer (TSan)**: Compile-time instrumentation (Clang/GCC `-fsanitize=thread`) that detects data races at runtime. Tracks the last read/write to every memory location along with the synchronization state. Reports the two conflicting accesses with full stack traces. Typically 5-15x runtime slowdown. - **Helgrind/DRD (Valgrind)**: Dynamic binary instrumentation tools for race and lock-order detection. No recompilation required but 20-50x slower than native execution. - **AddressSanitizer + LeakSanitizer (ASan/LSan)**: While primarily for sequential memory bugs (buffer overflow, use-after-free, leaks), these interact with concurrency — use-after-free on shared data is a common concurrency bug pattern. - **Lock-Order Analysis**: Tools track the order in which threads acquire locks. If thread A acquires lock1→lock2 and thread B acquires lock2→lock1, a potential deadlock is reported even if it hasn't occurred yet (static analysis of lock ordering). - **Model Checking**: Tools like CHESS (Microsoft) and CDSChecker systematically explore thread interleavings to find bugs that random testing would miss. Exhaustive for small programs; combinatorial explosion limits scalability. **GPU-Specific Tools** - **CUDA-MEMCHECK / Compute Sanitizer**: Detects race conditions in CUDA kernels, shared memory out-of-bounds, and misaligned accesses. - **NVIDIA Nsight Systems/Compute**: Profiling tools that visualize GPU kernel execution timeline, identifying synchronization bottlenecks and warp divergence. Parallel Debugging Tools are **the safety net for concurrent programming** — catching the non-deterministic, timing-dependent bugs that escape all other testing methods and would otherwise lurk in production code until they surface as rare, catastrophic, unreproducible failures.