parallel debugging gdb cuda

**Parallel Debugging and Correctness** tools enable **systematic identification and fixing of concurrency bugs (race conditions, deadlocks, synchronization errors) that are notoriously difficult to reproduce and diagnose in multi-threaded and GPU applications.** **CUDA-GDB Debugger for GPU Code** - **CUDA-GDB**: Integrated debugging environment for CUDA applications. Debugs both host (C/C++) and device (CUDA kernel) code simultaneously. - **Breakpoint Setting**: Set breakpoints on host or kernel code. Kernel breakpoints trigger per-thread or per-warp (all threads in warp break together). - **Variable Inspection**: Inspect host variables (standard gdb) and device variables (kernel local variables, shared memory, global memory). - **Thread Navigation**: Switch between host threads and kernel threads. Query thread registers, memory contents, execution state. **CUDA-GDB Capabilities and Limitations** - **Single-Stepping**: Step through kernel instructions (warp-level, not individual thread). All threads in warp advance together (synchronous execution). - **Conditional Breakpoints**: Break when thread_id == 5 AND block_id == 0. Enables targeted debugging of specific GPU threads. - **Print/Watch**: Monitor variable changes (memory access patterns). Track memory writes, identify corruption sources. - **Performance Impact**: Debugging 10-100x slower than normal execution. Suitable for small inputs, quick turnaround debugging. **Compute Sanitizer (cuda-memcheck)** - **cuda-memcheck**: Runtime memory debugging tool. Detects out-of-bounds accesses, uninitialized reads, memory leaks. - **Memcheck Detector**: Instruments kernels to track memory accesses. Every load/store checked against allocated memory ranges. - **False Positive Filtering**: Shared memory aliasing can trigger false positives (intentional pattern reuse). Configuration allows whitelisting. - **Overhead**: Instrumentation adds 5-50x slowdown. Suitable for correctness validation, not performance profiling. **Race Condition and Synchronization Detectors** - **Racecheck**: Detects data races (concurrent access to same memory location without synchronization). Uses dynamic analysis (instrument kernels) or static analysis (compile-time checks). - **Race Pattern**: Two threads access same memory location, at least one write, without synchronization (barrier, atomic). Pattern flagged as race. - **Shared Memory Races**: Racecheck detects shared memory races (common in GPU computing). Global memory races also detected (less common, often intentional with atomics). - **False Positives**: Properly synchronized code with complex synchronization patterns may trigger false alarms. Expert review necessary. **Initcheck and Other Detectors** - **Initcheck**: Detects unitialized shared memory accesses. Tracks which shared memory locations written. Reads to unwritten locations flagged. - **Synccheck**: Detects warp divergence, thread barriers within conditionals (can serialize execution). Identifies performance issues from incorrect synchronization. - **Combined Tools**: cuda-memcheck runs multiple detectors in single pass. Results aggregated, reported with source-line mapping. **Intel Inspector for CPU Parallelism** - **Inspector XE**: Detects data races, memory corruption, memory leaks in OpenMP/pthreads applications. - **Synchronization Analysis**: Tracks locks, barriers, semaphores. Identifies missing synchronization (race conditions), deadlocks. - **Memory Tracking**: Similar to cuda-memcheck. Monitors memory allocation, deallocation, accesses. - **Lightweight vs Detailed**: Light collection (minimal overhead, less info) for production; detailed collection for debugging (significant overhead). **Valgrind Helgrind for Multi-threaded Debugging** - **Helgrind Tool**: Memcheck for multi-threaded C/C++ programs. Detects races, synchronization issues via dynamic binary instrumentation. - **Happens-Before Graph**: Constructs synchronization graph. Race = two accesses violating happens-before relation (no synchronization path between them). - **False Positive Rate**: Significant false positive rate (~30-50%) due to conservative analysis. Manual verification of detected races required. - **Overhead**: 100-500x slowdown. Practical only for small test cases. **Parallel Correctness Workflows** - **Regression Testing**: Correctness tests run with multiple thread counts (2, 4, 8, etc.). Race conditions more likely with higher thread counts (higher contention). - **Stress Testing**: High contention artificially induced (tight loops, memory pressure). Amplifies race conditions, makes reproduction easier. - **Determinism**: Parallel programs inherently non-deterministic (thread scheduling random). Record-and-replay systems record execution path, enable deterministic replay for debugging. - **Symbol Debugging**: Build with debug symbols (-g compiler flag). Tools correlate memory addresses with source lines, enable source-level debugging. **Deadlock Detection and Avoidance** - **Deadlock Conditions**: Circular wait (Thread A holds lock L1 waiting for L2; Thread B holds L2 waiting for L1). All four Coffman conditions must be present. - **Static Analysis**: Code analysis identifying potential deadlock patterns (lock acquisition order violations). - **Dynamic Detection**: Runtime monitoring of lock wait-for graph. Cycle detection → deadlock alert. - **Prevention Strategies**: Enforce global lock ordering (if A then B then C). Timed locks (timeout instead of indefinite wait) recover from deadlocks.