locally typical sampling

**Locally typical sampling** is the **variant of typical sampling that applies typicality constraints at each decode step using local token distribution characteristics** - it emphasizes stepwise information-balance during generation. **What Is Locally typical sampling?** - **Definition**: Per-token decoding filter based on local entropy and surprisal deviation. - **Mechanism**: At each step, retain tokens near local typicality zone and sample from that subset. - **Local Adaptation**: Thresholding responds to immediate context uncertainty rather than global averages. - **Practical Role**: Used to stabilize open-ended generation without collapsing variety. **Why Locally typical sampling Matters** - **Stepwise Stability**: Prevents occasional low-quality jumps caused by local distribution spikes. - **Diversity Balance**: Maintains variation while avoiding extreme-token noise. - **Fluency Improvement**: Local typicality often preserves smoother sentence continuation. - **Prompt Robustness**: Adapts better across heterogeneous prompt styles and domains. - **Tuning Precision**: Provides fine-grained control over decoding behavior per position. **How It Is Used in Practice** - **Threshold Calibration**: Tune local typicality radius with domain-specific evaluation sets. - **Hybrid Pairing**: Combine with mild temperature scaling for broader stylistic control. - **Online Telemetry**: Track entropy and retained-token count across generation steps. Locally typical sampling is **a fine-grained entropy-guided decoding technique** - local typicality controls can improve consistency while preserving expressive variation.