Parallel Random Number Generation is the challenge of producing statistically independent, high-quality random sequences across multiple threads or processors simultaneously — requiring careful design to avoid correlations between streams that would invalidate Monte Carlo simulations, cryptographic applications, and stochastic algorithms while maintaining reproducibility for debugging.

Why Parallel RNG Is Hard

• Sequential RNG (single stream): Each output depends on previous state — inherently serial.
• Naively sharing one RNG across threads: Lock contention destroys performance.
• Giving each thread its own RNG with different seed: Risk of inter-stream correlation — statistical artifacts.

Parallel RNG Strategies

Counter-Based RNGs (Modern Best Practice)

• Philox, Threefry: Counter-based RNGs designed for parallel computing.
• Concept: output = encrypt(key, counter) — any counter value produces independent random output.
• Parallel: Each thread uses the same key, different counter range → perfectly independent, no state sharing.
• Reproducible: Given key + counter → always produces same output.
• Skip-ahead: Can jump to any position in O(1) — no need to generate preceding values.

Framework Implementations

Reproducibility in Deep Learning

• Set seed: torch.manual_seed(42) + torch.cuda.manual_seed_all(42).
• Deterministic mode: torch.use_deterministic_algorithms(True).
• Challenge: Multi-GPU training with different random augmentations per GPU — need independent but deterministic streams.
• Solution: Use SeedSequence.spawn() (NumPy) or separate Generator per data worker.

Statistical Testing

• TestU01 (BigCrush): Suite of statistical tests for RNG quality — 160+ tests.
• PractRand: Practical randomness testing suite.
• Good parallel RNG must pass both single-stream and inter-stream correlation tests.

Parallel random number generation is a foundational requirement for reproducible scientific computing — incorrect parallelization of RNG can silently introduce statistical artifacts that invalidate simulation results, making proper parallel RNG design essential for trustworthy Monte Carlo methods and stochastic training.