Concurrency in Python encompasses the techniques for executing multiple tasks simultaneously or in overlapping time periods — including threading (for I/O-bound tasks), asyncio (for high-concurrency I/O with cooperative scheduling), and multiprocessing (for CPU-bound tasks that bypass the GIL), with the choice between these approaches determined by whether the workload is I/O-bound or CPU-bound and the specific requirements for parallelism, memory sharing, and integration with async frameworks like those used in LLM API clients.

What Is Concurrency in Python?

• Definition: The ability to manage multiple tasks that make progress within overlapping time periods — concurrency (tasks interleave on one core) differs from parallelism (tasks execute simultaneously on multiple cores), though Python supports both through different mechanisms.
• GIL (Global Interpreter Lock): CPython's GIL allows only one thread to execute Python bytecode at a time — this means threading does NOT provide true parallelism for CPU-bound Python code, but it DOES allow parallel I/O operations because the GIL is released during I/O waits.
• Choosing the Right Tool: I/O-bound tasks (API calls, database queries, file I/O) benefit from threading or asyncio — CPU-bound tasks (data processing, model inference) require multiprocessing or external libraries (NumPy, PyTorch) that release the GIL during computation.

Concurrency Models

Async for LLM APIs

• Why Async: LLM API calls take 500ms-30s — async allows hundreds of concurrent requests on a single thread, maximizing throughput when calling OpenAI, Anthropic, or self-hosted models.
• AsyncOpenAI: The OpenAI Python client provides an async interface — await client.chat.completions.create() enables non-blocking API calls.
• asyncio.gather: Run multiple async calls concurrently — results = await asyncio.gather(*[call_api(p) for p in prompts]) processes all prompts in parallel.
• Rate Limiting: Use asyncio.Semaphore to limit concurrent requests — preventing API rate limit errors while maintaining high throughput.
• Streaming: Async streaming (async for chunk in response) enables real-time token delivery to users while other requests are processed concurrently.

When to Use Each Approach

• Threading: Simple I/O parallelism (downloading files, making a few API calls) — easy to use but limited scalability for thousands of connections.
• Asyncio: High-concurrency I/O (web servers, LLM API batching, websockets) — scales to thousands of concurrent connections on a single thread but requires async-compatible libraries.
• Multiprocessing: CPU-intensive work (data preprocessing, model inference without GPU) — true parallelism but higher memory overhead (each process gets its own memory space).
• External Libraries: NumPy, PyTorch, and other C-extension libraries release the GIL during computation — enabling true parallelism within threads for numerical workloads.

Concurrency in Python is the essential skill for building performant ML applications — choosing between threading, asyncio, and multiprocessing based on whether workloads are I/O-bound or CPU-bound, with async programming particularly critical for LLM applications that must efficiently manage hundreds of concurrent API calls and streaming responses.