token
**Tokenization in Large Language Models**
**What is Tokenization?**
Tokenization is the process of converting raw text into discrete units called tokens that language models can process. Unlike traditional word-based approaches, modern LLMs use subword tokenization to handle rare words and multiple languages efficiently.
**How BPE (Byte Pair Encoding) Works**
BPE starts with individual characters and iteratively merges the most frequent pairs:
1. Start with character vocabulary: ["h", "e", "l", "o", " ", "w", "r", "d"]
2. Find most common pair: ("l", "l") → merge to "ll"
3. Repeat until vocabulary size reached (typically 32K-100K tokens)
**Tokenization Algorithms Comparison**
| Algorithm | Used By | Key Feature |
|-----------|---------|-------------|
| BPE | GPT, Llama | Frequency-based merging |
| WordPiece | BERT | Likelihood-based merging |
| Unigram | T5, ALBERT | Probabilistic selection |
| SentencePiece | Llama, Mistral | Language-agnostic preprocessing |
**Practical Implications**
- **Token count affects cost**: API pricing is per token (1K tokens ≈ 750 words)
- **Context limits**: Max tokens per request (GPT-4: 128K, Claude: 200K)
- **Multilingual efficiency**: CJK characters often require 2-3 tokens per character
- **Code tokenization**: Programming symbols may use more tokens than expected
**Example Token Counts**
```python
"Hello, world!" → 4 tokens
"Semiconductor manufacturing" → 3 tokens
"人工智能" → 3 tokens (Chinese)
```
Understanding tokenization is crucial for prompt optimization and cost management.