tokenizers
**Hugging Face Tokenizers** is a **high-performance text tokenization library written in Rust with Python bindings that converts raw text into the token sequences that language models consume** — implementing BPE (Byte-Pair Encoding), WordPiece, Unigram, and SentencePiece algorithms at speeds of 1 GB of text in under 20 seconds, with character-level alignment tracking that maps every token back to its exact position in the original string.
**What Is Tokenizers?**
- **Definition**: A Rust-based library that handles the critical first step of any NLP pipeline — splitting raw text into subword tokens (pieces of words) that map to the vocabulary entries a language model was trained on, then converting those tokens to integer IDs for model input.
- **Rust Performance**: Written in Rust with Python bindings via PyO3 — tokenizing 1 GB of text takes under 20 seconds, compared to minutes or hours with pure Python implementations. The speed difference matters for preprocessing large training datasets.
- **Alignment Tracking**: Tokenizers maintains a character-to-token offset mapping — for every token, you can query exactly which characters in the original string it corresponds to. This is critical for NER tasks where you need to map token-level predictions back to character spans ("highlight the entity in the original text").
- **Parallelism**: Automatic multi-core parallelism for batch tokenization — tokenizing thousands of documents uses all available CPU cores without explicit threading code.
- **Training**: Train custom tokenizers from scratch on your own corpus — `tokenizer.train(files, trainer)` learns a vocabulary optimized for your domain (medical text, code, legal documents).
**Tokenization Algorithms**
| Algorithm | Used By | How It Works |
|-----------|---------|-------------|
| BPE (Byte-Pair Encoding) | GPT-2, GPT-4, LLaMA, Mistral | Iteratively merges most frequent character pairs |
| WordPiece | BERT, DistilBERT, Electra | Greedy longest-match from vocabulary |
| Unigram | T5, ALBERT, XLNet | Probabilistic subword selection |
| SentencePiece | LLaMA, T5, mBART | Language-agnostic BPE/Unigram on raw text |
**Key Features**
- **Pre-tokenization**: Configurable text splitting before subword tokenization — whitespace splitting, punctuation splitting, byte-level BPE, or custom regex patterns.
- **Normalization**: Unicode normalization (NFC, NFKC), lowercasing, accent stripping — applied consistently before tokenization.
- **Post-processing**: Automatic addition of special tokens ([CLS], [SEP], , ) — configured per model architecture.
- **Truncation and Padding**: Built-in truncation to max length and padding to batch length — with multiple truncation strategies (longest_first, only_first, only_second).
- **Fast Tokenizers in Transformers**: `AutoTokenizer.from_pretrained()` automatically loads the Rust-backed fast tokenizer when available — providing the same API as the Python tokenizer with 10-100× speed improvement.
**Hugging Face Tokenizers is the high-performance foundation that every Transformers model depends on** — converting raw text to model-ready token sequences at Rust speed with character-level alignment tracking, making it both the invisible workhorse of the Hugging Face ecosystem and an essential tool for teams training custom tokenizers on domain-specific corpora.