Sliding-window and sparse attention are techniques that cut the cost of the Transformer's attention by computing only a chosen subset of query-key pairs instead of all of them. Full self-attention scores every token against every other token, so both its compute and its KV-cache memory grow with the square of the sequence length — the wall that makes long context expensive. These methods replace the dense pattern with a structured one: a local window, a few global tokens, strided or random links, so that each token attends to far fewer others while the model still, layer by layer, propagates information across the whole sequence.\n\nSliding-window attention makes cost linear by attending only locally. Instead of letting a token see the entire history, sliding-window attention restricts each query to a fixed band of the most recent keys — a window of size w. Cost then scales as sequence length times w rather than length squared, and the KV cache need only hold the last w tokens per layer. Crucially, information still travels globally: just as stacked convolutions grow a receptive field, each layer lets a token reach w positions back, so after L layers the effective reach is about L times w. Mistral popularized this in a production LLM, pairing a modest window with enough depth to cover long documents.\n\nSparse patterns add global tokens to restore long-range reach. A pure window can miss important distant tokens, so sparse-attention models combine several fixed patterns. Longformer and BigBird keep a local window but designate a handful of global tokens — often special or task-relevant positions — that every token can attend to and that attend to everything, giving a short path between any two positions. BigBird adds random links and proves the combination is a universal approximator of full attention. The Sparse Transformer instead uses strided and block patterns aligned to the hardware. In every case the score matrix goes from fully dense to mostly empty, and the compute follows.\n\n| | Dense attention | Sliding window | Sparse (global+window) |\n|---|---|---|---|\n| Pairs scored | all n² | n·w (band) | n·w + global |\n| Cost | O(n²) | O(n·w) | ~O(n) |\n| Long-range path | direct | via depth (L·w) | via global tokens |\n| KV cache | all tokens | last w per layer | window + globals |\n| Risk | expensive | misses distant cues | pattern must fit task |\n| Examples | vanilla Transformer | Mistral, Longformer-local | Longformer, BigBird |\n\n``svg\n\n``\n\nIt is one of three levers on the attention bottleneck, and it composes with the others. Attention efficiency work attacks the quadratic in complementary ways: Flash Attention keeps the pattern dense but reorders the computation to avoid materializing the score matrix; MQA, GQA, and MLA shrink the bytes cached per token; sliding-window and sparse attention drop pairs outright. They stack — a model can run sparse attention with a Flash kernel and a compressed KV cache at once. The design cost is that a fixed sparsity pattern bakes in an assumption about which tokens matter, so a pattern tuned for local structure can miss the occasional long-range dependency the task actually needs, which is why global tokens and hybrid full/sparse layer schedules are common.\n\nRead sparse and sliding-window attention through a quant lens rather than a 'look at fewer tokens' lens: the number they move is the count of query-key pairs actually scored, dropping from n-squared toward n times a window plus a handful of global links, and both compute and KV memory follow that count directly. The levers are the window size and the global/random budget: widen the window or add globals and you recover more of dense attention's reach at higher cost, narrow them and you save more memory but risk severing a dependency the task relies on, so the design question is the smallest pattern whose paths still connect the tokens your data actually needs to relate.