Tokenization Subword Methods
# Tokenization & Subword Methods
## Introduction & Motivation
Tokenization: convert text to tokens. Subword methods like BPE, WordPiece. Applications: efficient encoding, vocabulary management.
Motivation: Convert raw text into model-ready tokens.
Applications: NLP preprocessing, vocabulary optimization, multilingual support.
---
## Core Concepts & Theory
### Byte-Pair Encoding
Merge frequent pairs.
### WordPiece
Probability-based merging.
### SentencePiece
Language-agnostic tokenization.
### Vocabulary Coverage
Balancing coverage and compression.
---
## Mathematical Formulation
BPE Merge:
$$ ext{merge}(a, b)
ightarrow ab$$
Frequency:
$$ ext{freq}(ab) = \sum_i \mathbb{1}[ab \in ext{text}_i]$$
Compression Ratio:
$$r = \frac{T_{ ext{chars}}}{T_{ ext{tokens}}}$$
---
## Advanced Theory & Extensions
### Morphological Tokenization
Handle complex morphology.
### Cross-Lingual Tokenization
Multilingual support.
### Adaptive Vocabularies
Task-specific optimization.
---
## Computational Considerations
BPE: O(V log V).
Encoding: O(T).
Decoding: O(T).
---
## Practical Implementation Strategies
### Vocabulary Size
Choose 30k-200k tokens.
### Merge Iterations
Control compression.
### Special Tokens
Handle edge cases.
---
## Benchmark Datasets & Evaluation
Compression: Bits per character.
Coverage: Out-of-vocabulary rate.
Language: Test on diverse languages.
---
## Key Challenges & Limitations
### Cross-Lingual
Different scripts, scripts.
### Domain Adaptation
Specialized vocabularies.
### Efficiency
Speed vs coverage tradeoff.
---
## Hyperparameter Tuning
Vocabulary size: 32k-256k.
Min frequency: 1-5.
Merge iterations: 1k-50k.
---
## Real-World Applications & Case Studies
NLP Models: BERT, GPT tokenization.
Multilingual: Handle many languages.
Specialized: Medical, legal text.
---
## Integration with Other Methods
Tokenization + embedding; + language model.
---
## Summary & Key Takeaways
Tokenization converts text efficiently.
Principles:
1. BPE: Byte-pair encoding.
2. WordPiece: Probability merging.
3. SentencePiece: Language-agnostic.
4. Vocabulary: Optimal size tradeoff.
5. Efficiency: Compression and coverage.
---
## Appendix: Practical Labs
### Lab 1: Byte-Pair Encoding
def bpe_encode(text, num_merges=100):
"""Apply BPE to text"""
# Start with characters
tokens = list(text)
for _ in range(num_merges):
# Find most frequent pair
pairs = {}
for i in range(len(tokens)-1):
pair = (tokens[i], tokens[i+1])
pairs[pair] = pairs.get(pair, 0) + 1
if not pairs:
break
# Merge most frequent
most_freq = max(pairs, key=pairs.get)
tokens = [most_freq if (tokens[i], tokens[i+1]) == most_freq
else tokens[i] for i in range(len(tokens))]
return tokens
tokens = bpe_encode("aaabdaaac", 10)
assert len(tokens) < 9
print(f"✓ BPE: {tokens}")### Lab 2: Vocabulary Statistics
def compute_vocab_stats(corpus, vocab_size=50000):
"""Compute vocabulary statistics"""
from collections import Counter
tokens = corpus.split()
counts = Counter(tokens)
coverage = sum(c for t, c in counts.most_common(vocab_size)) / len(tokens)
return coverage
corpus = "the cat sat on the mat the cat jumped on the mat"
coverage = compute_vocab_stats(corpus, 100)
print(f"✓ Vocabulary coverage: {coverage:.1%}")### Lab 3: Compression Ratio
def compression_ratio(original_text, tokenized):
"""Compute compression ratio"""
original_chars = len(original_text)
tokenized_chars = sum(len(t) for t in tokenized)
ratio = original_chars / tokenized_chars
return ratio
original = "hello world"
tokenized = ["hello", "world"]
ratio = compression_ratio(original, tokenized)
assert ratio > 1
print(f"✓ Compression ratio: {ratio:.2f}")### Lab 4: Out-of-Vocabulary Analysis
def oov_analysis(test_text, vocab):
"""Analyze out-of-vocabulary words"""
tokens = test_text.split()
oov_count = sum(1 for t in tokens if t not in vocab)
oov_rate = oov_count / len(tokens)
return oov_rate
vocab = {"hello", "world", "test"}
text = "hello world test new"
oov_rate = oov_analysis(text, vocab)
assert 0 <= oov_rate <= 1
print(f"✓ OOV rate: {oov_rate:.1%}")---