byte-level tokenization

Byte-level tokenization operates on raw bytes, enabling handling of any Unicode text without vocabulary gaps. **Core idea**: Instead of characters or subwords, tokenize at byte level (256 possible base tokens). Then apply BPE or other algorithms on bytes. **Universal coverage**: Any valid UTF-8 text can be tokenized, no unknown tokens ever. Handles emojis, rare scripts, code, everything. **Used by**: GPT-2, GPT-3, GPT-4 (byte-level BPE), CLIP text encoder. **Implementation**: Map bytes to printable characters for BPE processing, apply standard BPE on byte sequences. **Trade-off**: Non-ASCII characters use multiple bytes, so tokenization less efficient for non-English. CJK characters may use 3-4 bytes each. **Comparison**: Character-level has vocabulary per character (can be huge for Unicode), byte-level fixed at 256 base tokens. **Benefits**: No preprocessing needed, handles any input, robust to encoding issues. **Multilingual consideration**: Same model handles all languages but token efficiency varies significantly. **Modern standard**: Most production LLMs now use byte-level approaches for robustness.