python generator
A Python generator is a function that suspends instead of returning: the moment a function body contains `yield`, calling it stops executing anything and hands back a 192-byte object that holds the function's live stack frame — its variables, its instruction pointer, its place in every enclosing loop. That object is the data structure. Summing ten million squared integers through a list comprehension costs 409.09 MB of peak memory; the same sum through a generator expression costs 544 bytes, finishes 20.7 percent sooner, and returns its first value 22,175 times faster.
```svg
```
**A generator stores a position in a computation rather than the computation's output.** PEP 255 introduced `yield` in Python 2.2 and the mechanism has not changed since: when the compiler sees `yield` anywhere in a function body it sets a flag on the code object, and calling that function allocates a generator object instead of running anything. Each `next()` resumes the saved frame, runs until the next `yield`, and suspends again with every local variable exactly where it was. PEP 289 gave the same machinery a comprehension syntax in Python 2.4, PEP 342 added `send()` and `throw()` in Python 2.5 so a generator could receive as well as produce, and PEP 380 added `yield from` in Python 3.3 so generators could delegate to each other without a manual forwarding loop. The generator object is 192 bytes whether it will produce three values or a billion, because its size is set by the frame it holds, not by the sequence it describes.
**The memory argument is real, and it is larger than most people expect.** A list of one thousand integers occupies 8,856 bytes for the pointer array alone, before the integer objects themselves; scaled to ten million squared values the peak measured by `tracemalloc` is 409.09 MB. The generator expression over the identical computation peaks at 544 bytes, a factor of 752,012. Written as a scaling law the difference is not subtle, it is a change of complexity class,
$$M_{\text{list}} \;=\; n\,s_{\text{item}} \;+\; 8n \qquad\text{against}\qquad M_{\text{gen}} \;=\; 192\ \text{bytes}$$
so the list grows without bound in the length of the input while the generator does not grow at all. This is the difference between a script that streams a 40 GB fab log on a laptop and one that needs a machine with 64 GB of RAM to process the same file, and it is why every mature streaming interface in the ecosystem is generator-shaped: `pandas.read_csv` with a `chunksize`, the `csv` module's reader, PyTorch's `DataLoader`, and Hugging Face `datasets` in streaming mode all hand back iterators rather than materialised collections.
**Laziness changes latency far more dramatically than it changes throughput.** Building a list of five million squared values and taking the first element takes 182.50 ms, essentially all of it spent computing the 4,999,999 values that were never wanted; taking the first element from the equivalent generator takes 8.23 µs. That ratio of 22,175 is the entire argument for laziness in interactive and streaming work, and it compounds with early termination, because `next()`, `itertools.islice` and a `break` in a `for` loop all stop the producer permanently rather than after it has finished. A search that scans until it finds a match pays for the items it examined and nothing else, which is a property no eagerly built collection can offer at any size.
**Generators are not slower than lists, and on large inputs they are measurably faster.** The persistent folklore is that laziness buys memory at the cost of speed, and on this benchmark the opposite holds: summing the generator expression took 376.1 ms against 474.2 ms for the list comprehension, making the lazy version 20.7 percent quicker. The reason is allocation, not cleverness — the list version has to request, fill and eventually free 409.09 MB, and the allocator work plus the cache pressure of walking a 409 MB array costs more than the per-item suspension. The per-item numbers point the same way: a `yield` loop delivers an item in 19.28 ns while appending to a list and summing afterwards costs 30.34 ns. The honest caveat is that random access is a different question entirely, where a list index at 9.49 ns beats `next()` at 19.40 ns by 2.04 times, and no amount of laziness gives you element 4,000,000 without walking to it.
**The live frame is what makes a generator one-shot and stateful at the same time.** These two properties are usually taught as unrelated rules to memorise, and they are the same fact seen twice. Because the frame advances and is never rewound, a generator is exhausted after one pass — iterate it a second time and you get an empty result with no error, which is the single most common generator bug in production code and the reason a function returning a generator must never be consumed twice by a caller that does not know it. Because that same frame persists between resumptions, local variables survive across calls for free, so a running total, a parser state or a rolling window needs no class and no instance attributes. PEP 342 made the channel bidirectional: `send(2)` into a suspended generator makes the `yield` expression evaluate to 2, which is how a generator becomes a coroutine, and `close()` throws `GeneratorExit` at the suspension point and drops the frame, after which `gi_frame` reads as `None`.
**Composing generators into a pipeline keeps memory flat no matter how many stages you add.** Chaining a reader, a transformer and a filter over the same ten million items holds about 1 KB at peak, because each stage adds one 192-byte frame rather than one intermediate collection. The equivalent eager pipeline allocates a full copy at every stage, so three stages over 409 MB of data is upwards of a gigabyte of transient allocation that the garbage collector then has to reclaim. This is the structural argument for writing data plumbing as small generator functions and letting `itertools` supply the connectors, and it is why Dask, Apache Arrow's record-batch readers and most log-processing tooling expose batch iterators rather than whole-dataset handles.
**The per-item tax is the only cost you should actually be counting.** Everything above collapses to one comparison: 19.28 ns of suspension and resumption per item, weighed against what producing that item costs and what holding all the items would cost. On a stream of parsed log lines, decoded images or database rows the per-item work is microseconds to milliseconds and the suspension is invisible. On a tight numeric loop over an array already in memory, the suspension is the work, and the correct answer is neither a generator nor a list but NumPy, where the loop happens below the interpreter entirely. The failure mode worth naming is the one that quietly undoes all of it: wrapping `list()` around a generator restores the full 409.09 MB and the full 182.50 ms of latency, so a single defensive `list()` call in the middle of an otherwise lazy pipeline discards every advantage the pipeline was built for.
| Construction | Peak memory | First item | Total wall clock | Reuse |
|---|---|---|---|---|
| List comprehension | 409.09 MB | 182.50 ms | 474.2 ms | unlimited re-iteration |
| Generator expression | 544 bytes | 8.23 µs | 376.1 ms | one pass only |
| Generator function with `yield` | 192 bytes per object | immediate | 19.28 ns per item | one pass only |
| Three-stage generator pipeline | about 1 KB | immediate | one frame per stage | one pass only |
| `itertools.islice` over a generator | unchanged | immediate | stops the producer early | one pass only |
| `list()` wrapped around a generator | back to 409.09 MB | 182.50 ms | every saving discarded | unlimited re-iteration |
```flowchart
{ "rows": [
{ "type": "nodes", "items": [
{ "title": "def f() contains yield", "sub": "compiler flags the code object", "tone": "neutral" },
{ "title": "f() is called", "sub": "nothing in the body runs yet", "tone": "neutral" }
] },
{ "type": "arrow" },
{ "type": "group", "title": "A 192-byte generator object holding one live frame", "note": "size is set by the frame, never by the sequence", "cycle": true, "loop": "resume, run to the next yield, suspend", "items": [
{ "title": "next() resumes", "sub": "locals and instruction pointer restored", "tone": "green" },
{ "title": "yield suspends", "sub": "value out, frame preserved", "tone": "green" },
{ "title": "send() resumes with a value", "sub": "the yield expression evaluates to it", "tone": "green" },
{ "title": "The frame only moves forward", "sub": "which is why one pass is all you get", "tone": "orange" }
] },
{ "type": "arrow" },
{ "type": "nodes", "items": [
{ "title": "StopIteration or close()", "sub": "the frame is dropped, gi_frame is None", "tone": "orange" },
{ "title": "list() around it", "sub": "materialises everything, saving discarded", "tone": "orange" }
] }
] }
```
Read a generator through a *frame* lens rather than a *sequence* lens: the object you are holding is not a shorter list, it is a paused function, and every rule follows from what a paused function can and cannot do. Every difficulty in this space is that one substitution seen from a different side — 409.09 MB collapsing to 544 bytes is data being replaced by the instructions that would produce it, 182.50 ms collapsing to 8.23 µs is work not yet done rather than work done faster, one-shot exhaustion is a frame that has no reverse gear, `send()` is a paused function being handed an argument, and a stray `list()` is the paused function being forced to run to completion whether anyone needed the results or not. Decide whether you need the values or the recipe, and the choice between a list and a generator stops being a style question.