When a language model finishes a forward pass it does not hand you a word. It hands you a probability distribution over its entire vocabulary, and decoding is the policy you use to turn that distribution into the next token. The model is the same every time; the sampler is the dial you actually control at inference. Two people running the identical model can get a crisp deterministic answer or a wild creative riff purely by choosing different decoding settings.\n\nGreedy decoding takes the single most likely token at every step. It is fast, reproducible, and locally optimal, but it is also myopic: always grabbing the top token can walk the model into bland, repetitive, or degenerate loops because the globally best sentence sometimes starts with a locally second-best word.\n\nBeam search widens the search by keeping the k most probable partial sequences alive at once, extending all of them and pruning back to the top k each step. It reliably finds higher-probability full sequences and is the workhorse of machine translation and summarization, where there is roughly one correct answer. For open-ended generation it tends to produce safe, generic text and can collapse the beams onto near-duplicates.\n\nTemperature reshapes the distribution before you sample from it by dividing the logits by a scalar T inside the softmax. T below 1 sharpens the distribution and concentrates mass on the top tokens (more conservative); T above 1 flattens it and hands probability to the long tail (more diverse and more error-prone). T = 1 leaves the model's native distribution untouched, and T approaching 0 collapses back to greedy.\n\nTop-k sampling truncates the candidate set to the k highest-probability tokens, renormalizes, and samples from just those. It kills the long tail of absurd tokens, but a fixed k is a blunt instrument: when the model is confident, k is too generous, and when it is unsure, k is too stingy.\n\nTop-p (nucleus) sampling truncates by cumulative probability mass instead of by count — it keeps the smallest set of tokens whose probabilities sum to p (say 0.9) and samples from that. The candidate set breathes: it shrinks to a couple of tokens when the model is certain and expands to dozens when it is not, which is why top-p is the most widely used default for chat and creative generation. In practice teams stack a modest temperature with top-p and leave the rest alone.\n\n| Method | Determinism | Diversity | Best for | Failure mode |\n|---|---|---|---|---|\n| Greedy | Deterministic | None | Short factual answers, code | Repetition, blandness |\n| Beam search (k) | Deterministic | Low | Translation, summarization | Generic, near-duplicate beams |\n| Temperature (T) | Stochastic | Tunable | Global creativity knob | High T -> incoherence |\n| Top-k | Stochastic | Medium | Cutting the absurd tail | Fixed k mis-sizes the set |\n| Top-p / nucleus | Stochastic | Adaptive | Chat, open-ended text | Very high p -> drift |\n\n``svg\n\n``\n\nThe mistake most people make is treating decoding as an afterthought — a single "temperature" slider to nudge when output feels off. It is better understood as the interface between a fixed probabilistic model and the text you actually want. Greedy and beam search ask what is most probable; temperature, top-k, and top-p ask how much of the model's uncertainty should I let through, and in what shape. Read decoding through a shape-the-distribution lens rather than a pick-the-best-word lens, and every parameter stops being a magic number and becomes a deliberate statement about how much risk you want the model to take on each token.