Tokenization Subword Methods

# Tokenization & Subword Methods

## Introduction & Motivation

Tokenization: convert text to tokens. Subword methods like BPE, WordPiece. Applications: efficient encoding, vocabulary management.

Motivation: Convert raw text into model-ready tokens.

Applications: NLP preprocessing, vocabulary optimization, multilingual support.

---

## Core Concepts & Theory

### Byte-Pair Encoding

Merge frequent pairs.

### WordPiece

Probability-based merging.

### SentencePiece

Language-agnostic tokenization.

### Vocabulary Coverage

Balancing coverage and compression.

---

## Mathematical Formulation

BPE Merge:
$$ ext{merge}(a, b) ightarrow ab$$

Frequency:
$$ ext{freq}(ab) = \sum_i \mathbb{1}[ab \in ext{text}_i]$$

Compression Ratio:
$$r = \frac{T_{ ext{chars}}}{T_{ ext{tokens}}}$$

---

## Advanced Theory & Extensions

### Morphological Tokenization

Handle complex morphology.

### Cross-Lingual Tokenization

Multilingual support.

### Adaptive Vocabularies

Task-specific optimization.

---

## Computational Considerations

BPE: O(V log V).

Encoding: O(T).

Decoding: O(T).

---

## Practical Implementation Strategies

### Vocabulary Size

Choose 30k-200k tokens.

### Merge Iterations

Control compression.

### Special Tokens

Handle edge cases.

---

## Benchmark Datasets & Evaluation

Compression: Bits per character.

Coverage: Out-of-vocabulary rate.

Language: Test on diverse languages.

---

## Key Challenges & Limitations

### Cross-Lingual

Different scripts, scripts.

### Domain Adaptation

Specialized vocabularies.

### Efficiency

Speed vs coverage tradeoff.

---

## Hyperparameter Tuning

Vocabulary size: 32k-256k.

Min frequency: 1-5.

Merge iterations: 1k-50k.

---

## Real-World Applications & Case Studies

NLP Models: BERT, GPT tokenization.

Multilingual: Handle many languages.

Specialized: Medical, legal text.

---

## Integration with Other Methods

Tokenization + embedding; + language model.

---

## Summary & Key Takeaways

Tokenization converts text efficiently.

Principles:
1. BPE: Byte-pair encoding.
2. WordPiece: Probability merging.
3. SentencePiece: Language-agnostic.
4. Vocabulary: Optimal size tradeoff.
5. Efficiency: Compression and coverage.

---

## Appendix: Practical Labs

### Lab 1: Byte-Pair Encoding

def bpe_encode(text, num_merges=100):
 """Apply BPE to text"""
 # Start with characters
 tokens = list(text)
 
 for _ in range(num_merges):
 # Find most frequent pair
 pairs = {}
 for i in range(len(tokens)-1):
 pair = (tokens[i], tokens[i+1])
 pairs[pair] = pairs.get(pair, 0) + 1
 
 if not pairs:
 break
 
 # Merge most frequent
 most_freq = max(pairs, key=pairs.get)
 tokens = [most_freq if (tokens[i], tokens[i+1]) == most_freq 
 else tokens[i] for i in range(len(tokens))]
 
 return tokens

tokens = bpe_encode("aaabdaaac", 10)
assert len(tokens) < 9
print(f"✓ BPE: {tokens}")

### Lab 2: Vocabulary Statistics

def compute_vocab_stats(corpus, vocab_size=50000):
 """Compute vocabulary statistics"""
 from collections import Counter
 
 tokens = corpus.split()
 counts = Counter(tokens)
 
 coverage = sum(c for t, c in counts.most_common(vocab_size)) / len(tokens)
 return coverage

corpus = "the cat sat on the mat the cat jumped on the mat"
coverage = compute_vocab_stats(corpus, 100)
print(f"✓ Vocabulary coverage: {coverage:.1%}")

### Lab 3: Compression Ratio

def compression_ratio(original_text, tokenized):
 """Compute compression ratio"""
 original_chars = len(original_text)
 tokenized_chars = sum(len(t) for t in tokenized)
 ratio = original_chars / tokenized_chars
 return ratio

original = "hello world"
tokenized = ["hello", "world"]
ratio = compression_ratio(original, tokenized)
assert ratio > 1
print(f"✓ Compression ratio: {ratio:.2f}")

### Lab 4: Out-of-Vocabulary Analysis

def oov_analysis(test_text, vocab):
 """Analyze out-of-vocabulary words"""
 tokens = test_text.split()
 oov_count = sum(1 for t in tokens if t not in vocab)
 oov_rate = oov_count / len(tokens)
 return oov_rate

vocab = {"hello", "world", "test"}
text = "hello world test new"
oov_rate = oov_analysis(text, vocab)
assert 0 <= oov_rate <= 1
print(f"✓ OOV rate: {oov_rate:.1%}")

---