Self Attention

Self-attention lets every token in a sequence look at every other token and decide, for itself, which ones matter most to its own meaning right now — each token projects itself into a query, a key, and a value vector, and a token's new representation becomes a weighted sum of every token's value vector, with the weights set by how strongly that token's query matches each other token's key. Vaswani and colleagues introduced this exact formulation in their 2017 paper "Attention Is All You Need," and the paper's own go-to illustration still explains the mechanism better than any abstract description: in the sentence "The animal didn't cross the street because it was too tired," self-attention lets the token "it" compute a query that matches most strongly against the key for "animal," which is how the model resolves what "it" actually refers to — using nothing but learned vector similarity, with no explicit grammar rule telling it to do so.

The scaling factor isn't a cosmetic tweak — without it, self-attention breaks down as the vector dimension grows. Dot products between two vectors tend to grow larger in magnitude as the dimension they're computed in increases, and feeding large, widely varying dot products straight into softmax pushes the function into an extremely peaked region where almost all the probability mass concentrates on just one token and every gradient elsewhere collapses toward zero — the same saturation problem covered under vanishing-gradient, just appearing inside the attention computation itself. Dividing every dot product by the square root of the key dimension before the softmax keeps the values in a range where softmax still produces a well-behaved, gradient-friendly distribution regardless of how large the model's internal dimension gets.

Running multiple attention computations in parallel, called multi-head attention, lets different heads specialize in different kinds of relationships at once. The original Transformer used 8 attention heads in its base configuration and 16 in its larger variant, each with its own learned query, key, and value projections, so one head might end up tracking the kind of coreference resolution shown above while another tracks syntactic structure like subject-verb agreement, and a third tracks something with no clean linguistic name at all — the heads are never told what to specialize in, they simply end up dividing the relevance-tracking labor because gradient descent finds that division useful for minimizing the training loss.

Self-attention's compute and memory cost scales quadratically with sequence length, which is the single biggest engineering constraint on how much context a transformer can practically handle. Because every token computes a dot product against every other token, doubling the sequence length from 2,000 to 4,000 tokens doesn't double the attention computation — it quadruples it, since the number of token pairs grows with the square of the sequence length. This quadratic wall is exactly the problem that techniques like FlashAttention and various sparse-attention schemes exist to work around, trading some flexibility in which token pairs get full attention for a computation cost that scales closer to linearly with sequence length instead.

Self-attention's real innovation wasn't the idea of attending to relevant context — recurrent networks with attention, covered under encoder-decoder, could already do that — it was making every single position attend to every other position simultaneously, in one parallelizable operation, rather than one token at a time through a sequential bottleneck; that single architectural choice, trading recurrence for an all-pairs computation, is the specific trade that let transformers scale to context lengths and model sizes recurrent architectures never reached.