OOV (Out-of-Vocabulary) refers to words not in the models vocabulary, historically a major NLP challenge largely solved by subword tokenization. Traditional problem: Fixed word vocabularies could not handle unseen words, required UNK (unknown) token replacement, lost information. With subword tokenization: Words split into known subwords, virtually no true OOV. Cryptocurrency becomes crypto + curr + ency. When OOV still occurs: Character-level models with limited character set, very unusual Unicode, corrupted text. Handling strategies: Traditional: UNK replacement, spelling correction, stemming. Modern: Subword fallback to characters/bytes, byte-level tokenization guarantees no OOV. Rare token issues: While not technically OOV, rare subwords have poor embeddings due to limited training. Code and technical text: May contain identifiers and tokens underrepresented in training. Evaluation consideration: OOV rate used to measure vocabulary coverage on test sets. Modern status: Byte-level BPE and SentencePiece essentially eliminated OOV problem for text, shifting focus to rare token quality.
Explore 500+ Semiconductor & AI Topics
From EUV lithography to CUDA optimization — search the full knowledge base or chat with our AI assistant.