Long Context

Long context refers to a language model's ability to accept and use very large inputs in a single prompt, measured in tokens, so that whole books, codebases, transcripts or document collections can be placed in front of the model at once instead of being summarized or retrieved piece by piece. The context window, the maximum number of tokens a model can attend to in one pass, has grown by orders of magnitude in a few years: the original 2017 Transformer was demonstrated on sequences of a few hundred tokens, GPT-2 in 2019 handled 1,024, GPT-3 in 2020 handled 2,048, and by 2023 and 2024 commercial models advertised windows of 100,000 tokens (Anthropic's Claude, May 2023), 200,000 tokens (Claude 2.1, November 2023) and one million tokens (Google's Gemini 1.5 Pro, February 2024, with longer windows reported in research settings).

The core obstacle is that standard self-attention compares every token with every other token, so compute and memory for the attention scores grow with the square of the sequence length. Doubling the prompt quadruples the number of pairwise scores, which is why long windows were impractical for years. Two families of fixes made them feasible. Exact methods such as flash-attention reorganize the computation so the full score matrix is never written to slow memory, leaving the arithmetic quadratic but removing the memory blow-up, while the kv-cache stores each token's keys and values so that generation does not recompute them, at the price of a cache that grows linearly with context and often dominates GPU memory at long lengths. Approximate approaches such as sliding-window attention (used in Longformer by Beltagy, Peters and Cohan in 2020 and in Mistral 7B), sparse patterns, and state-space or hybrid architectures trade some global reach for cheaper scaling.

Position encoding is the second obstacle: a model trained on short sequences usually fails on longer ones because it has never seen those positions. Rotary position embeddings (RoPE, introduced by Su and colleagues in the 2021 RoFormer paper) encode position by rotating query and key vectors, and they became the default in models such as LLaMA. In 2023 several groups showed that a RoPE model could be stretched cheaply after pretraining: position interpolation (Chen et al., Meta) compresses the new, longer positions into the originally trained range and needs only a brief fine-tune, while NTK-aware scaling, YaRN (Peng et al., 2023) and related schemes adjust the rotation frequencies more carefully. Extending a window this way is a different thing from pretraining on long documents, and models stretched by position tricks alone often degrade in quality.

A window that fits the text is not the same as a model that uses it well, and the best-known evidence is the lost-in-the-middle effect. In the 2023 paper "Lost in the Middle: How Language Models Use Long Contexts," Nelson Liu and colleagues placed the answer-bearing passage at different positions in multi-document question answering and key-value retrieval tasks, and found accuracy was typically highest when the relevant information sat at the beginning or end of the input and dropped when it sat in the middle, a U-shaped curve, even for models explicitly built for long inputs. Later models narrowed this gap, but the lesson endures: position and distractors matter, and a longer prompt can make an answer harder to find rather than easier.

Needle-in-a-haystack tests, popularized in 2023 by Greg Kamradt, measure retrieval of one planted fact but are easy to over-read. A short, unrelated sentence is inserted at a chosen depth in long filler text and the model is asked to recall it; results are shown as a heat map of length against depth. Near-perfect scores became a standard claim in long-context model releases, yet the test checks little more than literal lookup of an out-of-place sentence. Harder suites such as RULER (Hsieh et al., 2024) add multi-needle retrieval, variable tracking, aggregation and question answering, and report that many models claiming very large windows have an effective length well below the advertised one on those tasks.

Long context competes with retrieval rather than replacing it, and each has costs. Stuffing a whole corpus into the prompt avoids chunking and reranking errors and lets the model reason across documents, but every request pays to process all those tokens, latency to the first output token rises with prompt length, providers typically charge per input token, and accuracy can fall as irrelevant material accumulates. Retrieval-augmented generation keeps prompts short and cheap and scales to corpora far larger than any window, but can miss evidence that was never retrieved. Prompt caching, which reuses the processed prefix across calls, makes repeated long prompts much cheaper. A common misconception is that a larger window means the model remembers more: it only sees what is in the current prompt, nothing persists afterward, and quality is bounded by how well attention actually distributes over those tokens.

Read long context as a bigger desk rather than a better memory: more material can be spread out at once, but the cost grows quadratically, the model's attention is uneven across the pile, and only measured retrieval and reasoning tests show how much of the window is truly usable.