LLM Decoding Strategies are the algorithms that determine how tokens are selected from a language model's probability distribution during text generation — ranging from deterministic methods like greedy and beam search to stochastic approaches like nucleus (top-p) sampling and temperature scaling, and advanced methods like contrastive decoding that exploit differences between strong and weak models, where the choice of decoding strategy profoundly affects output quality, diversity, coherence, and factuality.

Decoding Methods Overview

Temperature Scaling

def temperature_sample(logits, temperature=1.0):
    """Lower temp = more confident/deterministic
       Higher temp = more random/creative"""
    scaled = logits / temperature
    probs = softmax(scaled)
    return sample(probs)

# temperature=0.0: Greedy (argmax)
# temperature=0.3: Focused, factual responses
# temperature=0.7: Balanced (common default)
# temperature=1.0: Original distribution
# temperature=1.5: Very creative, sometimes incoherent

Top-p (Nucleus) Sampling

def top_p_sample(logits, p=0.9):
    """Sample from smallest set of tokens with cumulative prob >= p"""
    sorted_probs, sorted_indices = torch.sort(softmax(logits), descending=True)
    cumulative_probs = torch.cumsum(sorted_probs, dim=-1)
    # Remove tokens with cumulative probability above threshold
    sorted_probs[cumulative_probs > p] = 0
    sorted_probs[0] = max(sorted_probs[0], 1e-8)  # keep at least top-1
    # Renormalize and sample
    sorted_probs /= sorted_probs.sum()
    return sample(sorted_probs)

# p=0.1: Very focused (often 1-3 tokens)
# p=0.9: Standard (typically 10-100 tokens in nucleus)
# p=1.0: Full distribution (= temperature sampling only)

Contrastive Decoding

Idea: Amplify what a STRONG model knows that a WEAK model doesn't

score(token) = log P_large(token) - α × log P_small(token)

Intuition:
  - Both models predict common tokens similarly → low contrast
  - Large model uniquely confident about factual/coherent tokens → high contrast
  - Result: Suppresses generic/repetitive tokens, promotes informative ones

Effect: Significantly reduces hallucination and repetition

Min-p Sampling

def min_p_sample(logits, min_p=0.05):
    """Keep tokens with probability >= min_p × max_probability"""
    probs = softmax(logits)
    threshold = min_p * probs.max()
    probs[probs < threshold] = 0
    probs /= probs.sum()
    return sample(probs)

# Advantage over top-p: Adapts to distribution shape
# Confident prediction (one 90% token): min-p keeps very few tokens
# Uncertain prediction (many ~5% tokens): min-p keeps many tokens

Recommended Settings by Task

Repetition Penalties

• Frequency penalty: Reduce probability proportional to how often token appeared.
• Presence penalty: Fixed reduction if token appeared at all.
• Repetition penalty (multiplier): Divide logit by penalty factor for repeated tokens.
• These fix the degenerate repetition common in greedy/beam search.

LLM decoding strategies are the often-overlooked lever that dramatically affects generation quality — the same model can produce boring, repetitive text with greedy decoding or creative, diverse text with tuned sampling, and advanced methods like contrastive decoding can reduce hallucination by 30-50%, making decoding configuration as important as model selection for production AI systems.