frequency penalty

**Frequency penalty** is the **decoding adjustment that lowers token likelihood proportionally to how often the token has already appeared in the generated output** - it controls overuse of repeated vocabulary patterns. **What Is Frequency penalty?** - **Definition**: Token-level penalty scaled by occurrence count across generated text. - **Difference from Repetition Penalty**: Frequency penalties increase with count, not only prior occurrence presence. - **Effect Pattern**: Common words remain possible but become progressively less favored after repeated use. - **Usage Context**: Applied in stochastic or deterministic decoding to improve lexical variety. **Why Frequency penalty Matters** - **Lexical Diversity**: Prevents excessive reuse of the same terms and phrases. - **Readability**: Produces smoother prose with less mechanical repetition. - **Style Control**: Helps enforce richer expression in creative and explanatory outputs. - **Degeneration Prevention**: Reduces collapse into repeated token cycles. - **User Preference**: More varied wording generally improves perceived response quality. **How It Is Used in Practice** - **Penalty Tuning**: Set moderate values to balance variation against terminological precision. - **Task-Specific Profiles**: Use lighter penalties for technical QA where key terms must repeat. - **Combined Controls**: Coordinate with temperature and repetition penalties for stable behavior. Frequency penalty is **an effective tool for vocabulary-variation control in decoding** - frequency-aware penalties reduce monotony while maintaining output coherence.