unk token

**UNK token** is the **special placeholder token used when input text contains symbols or words not represented by the tokenizer vocabulary** - it provides fallback handling for out-of-vocabulary content. **What Is UNK token?** - **Definition**: Reserved token that substitutes unknown pieces during encoding. - **Trigger Condition**: Appears when tokenizer cannot map text span to known tokens. - **Encoding Role**: Prevents encoding failure by preserving sequence structure with placeholder symbols. - **Model Context**: More frequent in legacy or word-level tokenizers than modern subword systems. **Why UNK token Matters** - **Robustness**: Ensures inference continues even with rare or malformed input text. - **Coverage Signal**: High UNK rates indicate vocabulary mismatch with deployment domain. - **Quality Impact**: Too many UNK tokens reduce semantic fidelity and downstream accuracy. - **Monitoring Value**: UNK frequency is a useful health metric for tokenizer maintenance. - **Migration Guidance**: Persistent UNK problems often motivate tokenizer retraining or adaptation. **How It Is Used in Practice** - **Rate Tracking**: Monitor UNK occurrence by language, endpoint, and document source. - **Domain Expansion**: Retrain tokenizer on representative corpora to reduce OOV fragments. - **Input Sanitization**: Normalize corrupted characters and unsupported symbols before encoding. UNK token is **a fallback safety mechanism in tokenization pipelines** - controlling UNK frequency is essential for stable model understanding quality.