Home Knowledge Base Top K Sampling

Top-k sampling is a text generation strategy that restricts token selection to the k highest-probability tokens from the model's output distribution, then samples randomly among them, balancing quality and diversity. Mechanism: (1) compute logits for all vocabulary tokens, (2) apply softmax to get probabilities, (3) keep only top k tokens (zeroing others), (4) renormalize remaining probabilities, (5) sample from truncated distribution. Common values: k=40-100 for balanced generation, k=1 equivalent to greedy decoding. Comparison: greedy (always pick highest—deterministic, repetitive), pure sampling (sample from full distribution—diverse but incoherent), top-k (truncate then sample—compromise). Limitations: fixed k ignores distribution shape—when model is confident (peaked distribution), k tokens may include unlikely tokens; when uncertain (flat distribution), k may exclude reasonable options. Improved alternatives: top-p/nucleus sampling (dynamic cutoff based on cumulative probability), typical sampling (sample from tokens with typical information content). Temperature interaction: apply temperature scaling before top-k (T<1 sharpens, T>1 flattens distribution). Implementation: available in all major frameworks (transformers, llama.cpp). Use cases: creative writing, dialogue generation, and any application needing controlled randomness. Foundation decoding technique for language model inference.

top k samplingtruncaterandom

Related Topics

Explore 500+ Semiconductor & AI Topics

From EUV lithography to CUDA optimization — search the full knowledge base or chat with our AI assistant.