bos token
**BOS (Beginning of Sequence) Token** is a **special token that marks the start of an input sequence in transformer models** — providing a consistent initial context that enables the model to recognize sequence boundaries, initialize its hidden state from a known starting point, and distinguish between multiple independent sequences within the same batch, with different model families using different BOS conventions ([CLS] in BERT, in LLaMA, <|endoftext|> in GPT-2).
**What Is the BOS Token?**
- **Definition**: A reserved token in the model's vocabulary that is prepended to every input sequence — it occupies position 0 in the sequence, receives the first positional embedding, and serves as an explicit signal that a new sequence is beginning.
- **Sequence Boundary**: In batched processing where multiple sequences are packed together, the BOS token tells the model where one sequence ends and another begins — without it, the model cannot distinguish between a continuation of the previous sequence and the start of a new one.
- **Initial Context**: The BOS token provides a consistent, learned starting representation — the model's first attention computation has a known anchor point rather than starting from an arbitrary token.
- **Classification Token**: In encoder models like BERT, the BOS token ([CLS]) serves double duty — its final hidden state is used as the sequence-level representation for classification tasks (sentiment, NLI, similarity).
**BOS Tokens Across Model Families**
| Model | BOS Token | Token ID | Also Used As |
|-------|----------|---------|-------------|
| BERT | [CLS] | 101 | Classification head input |
| GPT-2 | <|endoftext|> | 50256 | Both BOS and EOS |
| LLaMA / LLaMA 2 | | 1 | Sequence start only |
| T5 | (none explicit) | N/A | Uses task prefix instead |
| Mistral | | 1 | Same as LLaMA convention |
| Gemma | | 2 | Sequence start |
| ChatML format | <|im_start|> | varies | Message boundary |
**BOS Token Functions**
- **Sequence Initialization**: Provides the first position embedding and initial attention anchor — the model learns what "the beginning of a sequence looks like" during training.
- **Batch Boundary Detection**: In continuous batching and packed sequences, BOS tokens mark where new sequences start — critical for attention masking to prevent cross-sequence attention leakage.
- **Classification Pooling**: In BERT-style models, the [CLS] token's final representation aggregates information from the entire sequence through self-attention — used as input to classification heads.
- **Chat Template Markers**: In chat models, BOS-like tokens (<|im_start|>, [INST]) mark the beginning of each message turn — enabling the model to distinguish between system, user, and assistant messages.
**BOS tokens are the explicit sequence initialization markers that transformer models depend on for boundary detection and consistent starting context** — a small but essential piece of the tokenization protocol that ensures models correctly process the beginning of every input sequence across batched inference, chat conversations, and classification tasks.