Autoregressive Language Model

An autoregressive model predicts each element of a sequence conditioned on every element that came before it, and the whole approach rests on one piece of exact, lossless algebra: the probability of an entire sequence can always be rewritten as the product of each token's probability given everything prior to it, by the chain rule of probability — no approximation involved, just a restatement of the same joint probability one token at a time. The term comes from classical statistics, where Udny Yule's 1927 and Gilbert Walker's 1931 work on autoregressive time-series models predicted each value in a series from its own preceding values; the modern neural version extended the same idea first to images, with van den Oord and colleagues' 2016 PixelRNN and PixelCNN generating one pixel at a time conditioned on every pixel already drawn, and then to text, with Radford and colleagues' 2018 GPT paper applying the identical factorization to predict each word from everything written so far.

Start with a context (possibly empty)
whatever tokens exist so far
Model outputs a probability distribution
over every possible next token, given the current context
One token is sampled or selected
from that distribution — see softmax and next-token-prediction
New token appended to the context
the sequence is now one token longer

The chain-rule factorization is why an autoregressive model only ever needs to look backward, which is exactly why causal, masked self-attention exists as a distinct variant. Because each token's probability is defined purely in terms of the tokens before it, a decoder generating text has no legitimate reason to attend to tokens that haven't been generated yet — allowing that would let the model "cheat" by seeing the answer during training — so autoregressive transformer decoders mask out all future positions in self-attention, restricting every token's query to only the keys and values of itself and everything earlier. This is the specific architectural difference between a GPT-style autoregressive decoder and the bidirectional encoder covered under encoder-decoder, which has no such restriction because it isn't predicting one token at a time at all.

One Exact Factorization, Token by Token the chain rule applied to "The cat sat down" — no approximation at any step P(The, cat, sat, down) = P(The) × P(cat | The) × P(sat | The, cat) × P(down | The, cat, sat) Step 1 context: (empty) predicts: "The" Step 2 context: "The" predicts: "cat" Step 3 context: "The cat" predicts: "sat" Step 4 context: "The cat sat" predicts: "down" Each step only ever sees tokens already generated — never anything to its right. This left-to-right restriction is exactly what causal (masked) self-attention enforces. Multiply all four factors back together and you recover the original joint probability exactly. No information is lost or approximated — this decomposition is pure probability algebra.

Training and generation feed the model different things, and the gap between them — exposure bias — is a known, named failure mode. During training, the model predicts each token conditioned on the *true* preceding tokens from the training data, a setup called teacher forcing because the ground truth is fed in regardless of what the model itself would have predicted; at actual generation time, there is no ground truth left to feed in, so the model has to condition on its own previously generated tokens instead, and a single early mistake can compound, since every later prediction is now conditioned on a context containing that error. Bengio and colleagues' 2015 paper "Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks" was an early, direct attempt to narrow this exact train-versus-generate mismatch by occasionally feeding the model's own predictions back in during training instead of always using ground truth.

Virtually every major large language model in production today — GPT, Llama, Claude, Gemini — is autoregressive, which makes the small but growing body of non-autoregressive alternatives worth naming for contrast. Diffusion-based language models, an active research direction extending image diffusion techniques to text, generate by iteratively refining an entire sequence in parallel rather than committing to one token at a time in strict left-to-right order, trading the autoregressive model's simple, exact factorization for the possibility of generating faster by not waiting on a sequential chain — though as of this writing, autoregressive generation remains the dominant, most thoroughly proven approach at the scale frontier labs actually deploy.

The autoregressive factorization is simultaneously the most mathematically unremarkable fact about how these models work — it's just the chain rule, no model-specific cleverness involved — and the single architectural decision with the widest downstream consequences: it's why decoders mask future tokens, why training and inference feed the model different kinds of context, why generation is inherently sequential rather than parallel, and why an error made early in a generated sequence has no way to get corrected except by the model recovering from it on its own.

Take autoregressive language model further

Ask the copilot about this term, or have our engineers assess it against your process.

Ask CFSGPT about this → Request a technical assessment → Open the CFS app