wordpiece

WordPiece is a subword tokenization algorithm used by BERT that builds vocabulary based on likelihood. **Algorithm**: Similar to BPE but selects merges by maximizing training data likelihood rather than raw frequency. Prefers merges that increase corpus probability. **Training**: Initialize with characters, iteratively merge pairs that maximize likelihood, continue until vocabulary size. **Format**: Continuation subwords marked with ## prefix (e.g., playing becomes play + ##ing). **Advantages**: Linguistically motivated merges, handles OOV through subword splits, efficient vocabulary usage. **Used by**: BERT, DistilBERT, ELECTRA, and their variants. **Vocabulary size**: BERT uses 30,522 tokens, including special tokens. **Comparison to BPE**: BPE uses frequency, WordPiece uses likelihood. Results often similar in practice. **Special tokens**: CLS, SEP, MASK, PAD, UNK for BERT-specific purposes. **Implementation**: HuggingFace tokenizers, original Google implementation. **Legacy**: While GPT-family uses BPE, BERT-family models continue using WordPiece, making it important for understanding encoder models.