next token prediction

Next token prediction is the fundamental training objective for autoregressive language models (like GPT), where the model learns to maximize the likelihood of the next token $x_t$ given the sequence of previous tokens $x_{1:t-1}$. Causal masking: the attention mechanism is masked (upper triangular matrix set to $-infty$) to prevent the model from "peeking" at future tokens. Self-supervised: no human labeling required; vast amounts of text can comprise the dataset. Probability distribution: the output is a probability distribution over the vocabulary; during inference, tokens are sampled from this distribution. Teacher forcing: during training, the model is fed the ground truth previous tokens, not its own specific predictions. Efficiency: allows parallel computation of loss for all tokens in a sequence simultaneously (unlike RNNs). Scaling: this simple objective, when scaled with data and compute, leads to emergent reasoning capabilities. Limitations: lacks planning or lookahead; "hallucinations" can propagate if an initial error is made. Next token prediction remains the dominant paradigm for generative AI.