greedy decoding

Every time a language model finishes a forward pass it hands you not a word but a *probability distribution* over all possible next tokens, and a decoding strategy is the rule that turns that distribution into actual text. Greedy decoding and beam search are the two *deterministic* strategies — they try to find the most probable output rather than rolling dice — and the difference between them is simply how much of the enormous tree of possible continuations they can afford to explore before committing. Greedy looks one step ahead and grabs the best token; beam search keeps several candidate sentences alive at once. Understanding when each wins, and why both lose to random sampling for creative text, comes down to one question: are you searching for *the* correct answer, or generating *an* interesting one?\n\n**Greedy decoding takes the single most likely token at every step — fast, but myopic.** At each position it computes the argmax of the distribution, appends that one token, and moves on, never reconsidering. It is as cheap as decoding gets and fully deterministic, but it is locally greedy in the literal sense: the highest-probability *first* token can lead into a corner where every continuation is poor, and greedy has no way to back out. Because it always chooses the safest token it is also prone to bland, repetitive loops — the model keeps picking the same high-probability phrase because nothing ever forces it off the well-worn path.\n\n**Beam search keeps the top-k partial sequences alive, trading compute for a better global score.** Instead of one running sentence it maintains k of them (the *beam width*). At every step it expands all k candidates by every possible next token, scores each extended sequence by its cumulative log-probability, and keeps only the best k — pruning the rest. This lets it recover from a locally attractive but globally bad early choice, approximating a search for the single highest-probability *whole* sequence rather than the greedy token-by-token path. Two details matter: setting k=1 reduces beam search exactly to greedy, and because longer sequences accumulate more negative log-probs, beam search needs *length normalization* or it will systematically prefer short, truncated outputs.\n\n**For open-ended generation both lose to sampling, because the most probable text is often the most boring.** This is the counterintuitive lesson: pushing beam width higher finds ever-higher-probability sequences, and those sequences get *worse* — generic, repetitive, degenerate ("I don't know. I don't know. I don't know."). The highest-likelihood continuation of a creative prompt is a safe cliché, not an interesting completion. So beam search shines on *closed-ended* tasks where a correct answer exists and fidelity matters — machine translation, speech recognition, short summarization — while *open-ended* generation (chat, story writing) uses stochastic sampling with temperature and top-p to inject the diversity that maximizing probability destroys. This is why modern LLM chat interfaces sample rather than beam-search.\n\n| Strategy | How it picks tokens | Best for |\n|---|---|---|\n| Greedy | argmax, one token, no lookahead | Fast baselines; short deterministic outputs |\n| Beam search (k>1) | Keep top-k sequences by cumulative log-prob | Translation, ASR, summarization |\n| Beam, large k | Finds highest-probability whole sequence | Diminishing/negative returns — text gets bland |\n| Sampling (temp, top-p) | Draw randomly from the distribution | Open-ended, creative, conversational text |\n\n```svg\n\n```\n\nThe unhelpful way to think about greedy versus beam search is as a contest with a winner — as if beam search were simply the smarter, better version you use when you can afford it. The useful way is to see both as *search over a tree of possible sentences*, where greedy explores one branch and beam explores k, so beam finds higher-probability whole sequences precisely because it can abandon a tempting but doomed early choice. The twist is that higher probability is only the right target when there is a correct answer to converge on; for open-ended generation the most probable sentence is the most forgettable one, which is why chat models sample instead. Read the greedy-vs-beam-vs-sampling choice through a what-am-I-actually-optimizing lens — fidelity to one right answer, or diversity across many good ones — rather than a which-decoder-is-best lens, and the strategy you should reach for stops being a default and becomes a direct consequence of the task in front of you.