thread
**Threads vs Processes in Python AI** is the **critical concurrency architecture decision governed by Python's Global Interpreter Lock (GIL)** — threads are correct for I/O-bound LLM API calls and database queries, while multiprocessing is necessary for CPU-bound operations like tokenization, preprocessing, and data augmentation that need true parallelism.
**What Is the Python GIL?**
- **Definition**: The Global Interpreter Lock is a mutex in CPython that prevents multiple threads from executing Python bytecode simultaneously — only one thread can hold the GIL and run Python code at any moment, even on a multi-core CPU.
- **Why It Exists**: Python's memory management (reference counting) is not thread-safe. The GIL prevents race conditions in the interpreter's internal state without requiring fine-grained locking on every object.
- **Critical Impact**: A 32-core server running a pure Python computation in 8 threads achieves no speedup vs 1 thread — all 8 threads time-share a single core's worth of Python execution due to the GIL.
- **When GIL Is Released**: The GIL IS released during I/O operations (network, disk), C extension code (NumPy, PyTorch CUDA kernels), and time.sleep() — enabling genuine concurrency for I/O-bound tasks.
**Threads: When to Use**
Python threads ARE effective when:
- **Network I/O**: Calling OpenAI API, fetching from vector DB, querying Redis. Thread releases GIL while waiting for network — other threads run.
- **File I/O**: Reading documents from disk, writing logs. GIL released during kernel I/O.
- **C Extension Work**: NumPy and PyTorch operations release the GIL — multiple threads can run CPU-bound NumPy array operations in parallel.
- **Wait-heavy workloads**: Threads holding the GIL for microseconds between I/O calls — overhead is negligible.
from concurrent.futures import ThreadPoolExecutor
import httpx
def call_api(prompt: str) -> str:
# Network I/O — GIL released while waiting
return httpx.post(LLM_API_URL, json={"prompt": prompt}).json()
with ThreadPoolExecutor(max_workers=20) as executor:
results = list(executor.map(call_api, prompts)) # 20 concurrent API calls
**Processes: When to Use**
Python multiprocessing IS required when:
- **CPU-bound preprocessing**: Tokenization, text cleaning, BPE encoding, audio transcoding — pure Python CPU work not releasing GIL.
- **Data augmentation**: Image transforms (resize, crop, normalize) in pure Python — needs real parallelism.
- **Parallelizing training loops**: Each process gets its own Python interpreter and GIL — true parallel execution.
- **DataLoader workers**: PyTorch DataLoader uses multiprocessing for workers — each worker process independently loads and preprocesses batches.
from torch.utils.data import DataLoader
# num_workers > 0 → multiprocessing, each worker is a separate process
dataloader = DataLoader(dataset, batch_size=32, num_workers=8)
from multiprocessing import Pool
def tokenize_document(doc: str) -> list[int]:
return tokenizer.encode(doc) # CPU-bound — needs true parallelism
with Pool(processes=8) as pool:
token_lists = pool.map(tokenize_document, documents)
**The Correct Concurrency Model for AI Systems**
| Task | Model | Why |
|------|-------|-----|
| LLM API calls | Async/threads | I/O bound — GIL released |
| Vector DB queries | Async/threads | I/O bound — GIL released |
| Image augmentation | Multiprocessing | CPU bound — needs true parallelism |
| Tokenization | Multiprocessing | CPU bound |
| PyTorch CUDA training | Threads OK or async | CUDA releases GIL |
| JSON parsing | Multiprocessing | CPU bound |
| DataLoader prefetching | Multiprocessing (built-in) | CPU preprocessing |
**Memory Model Differences**
**Threads**: Shared memory space — all threads see the same Python objects. Fast to create (~microseconds), low memory overhead, but requires locks for shared mutable state.
**Processes**: Separate memory spaces — each process has its own copy of all data. Slow to create (fork: ~milliseconds), high memory overhead (copy-on-write until modified), but completely isolated — crashes do not propagate.
**IPC (Inter-Process Communication)**:
- Queue/Pipe: Pass data between processes via serialization (pickle).
- Shared Memory (multiprocessing.shared_memory): Zero-copy sharing of arrays between processes.
- Memory-mapped files: Share large datasets across processes.
**GIL in Python 3.13+**
Python 3.13 introduces optional free-threading (GIL-free) mode — early support, not yet production-ready for most AI workloads. The GIL remains the default. This will eventually change the threads-vs-processes calculus for CPU-bound Python code, but for now the rules above apply.
The threads vs processes decision is **the architectural foundation of Python AI system performance** — choosing threads for I/O-bound LLM API calls enables efficient concurrency, while choosing multiprocessing for CPU-bound preprocessing enables the true parallelism that multi-core hardware provides, together ensuring that neither the network nor the CPU becomes an unnecessary bottleneck.