Token is an integer-indexed unit emitted by a tokenizer and consumed or generated by a language model. Tokens determine sequence length, embeddings, attention and KV-cache cost, context limits, training batches, latency, and usage accounting. A token may represent a word, subword, byte sequence, character, whitespace pattern, punctuation mark, control symbol, image patch, audio code, or multimodal placeholder; it is not inherently a word. A production definition states the base model and revision, tokenizer and vocabulary, context and output limits, numerical precision, data provenance, objective, trainable state, inference runtime, tool or retrieval boundary, evaluation population, latency and cost target, failure policy, and reproducibility artifacts. Similar labels can hide materially different implementations, so exact interfaces and assumptions belong in the contract. A tokenization contract includes tokenizer files and hash, normalization, pre-tokenization, algorithm, merge or model vocabulary, byte fallback, special-token IDs, added tokens, padding, truncation, chat template, and decode behavior.

Architecture, representation, and operating mechanism. Text is normalized and segmented, tokenization maps pieces to vocabulary IDs, embedding lookup converts IDs into vectors, position information is added, the model predicts a distribution over vocabulary IDs, and decoding samples IDs that the tokenizer converts back to bytes or text. Vocabulary size trades sequence length against embedding/output-matrix size and rare-piece behavior. English prose often averages around a fraction of a word to roughly a word per token depending on tokenizer and domain, but code, numbers, whitespace, and languages differ sharply; no fixed conversion is reliable. Word, character, byte, BPE, WordPiece, Unigram/SentencePiece, byte-level BPE, and multimodal tokenizers have different coverage and segmentation. Special tokens mark roles, boundaries, tools, padding, images, or control state and must not collide with user text. The complete stack includes input normalization, tokenization, embeddings, Transformer blocks, attention and KV state, output decoding, adapters or post-training weights, retrieval and tools where used, orchestration, policy controls, telemetry, and artifact storage. Data, control, and trust boundaries should remain visible instead of being collapsed into a single model call. Evaluation keeps task quality beside factuality, calibration, robustness, safety, subgroup behavior, context utilization, throughput, time to first token, inter-token latency, tail latency, memory, bandwidth, accelerator utilization, energy, and cost. Controlled comparisons hold prompts, sampling, data, model, hardware, concurrency, and judge protocol fixed and report uncertainty across repeated runs.

Implementation, serving infrastructure, and failure modes. Pin tokenizer with the checkpoint, test encode-decode round trips, reserve and escape special tokens, count after applying chat templates, avoid truncating critical suffixes, mask padding and prompt loss correctly, and stream only valid decoded byte sequences. Token count scales attention, KV cache, activation memory, training FLOPs, decode iterations, and communication. Vocabulary projection and sampling touch large matrices; tokenizer CPU performance and host-GPU scheduling can bottleneck high-throughput serving. Using the wrong tokenizer produces plausible IDs with wrong meaning, special-token injection crosses roles, Unicode normalization changes text, byte sequences decode incompletely, word-based cost estimates fail, truncation removes instructions, or vocabulary resize misaligns embeddings. Implementation starts with a small explicit reference, typed schemas, deterministic fixtures, versioned prompts and templates, and traceable input-output examples. Production adds batching, streaming, mixed precision, compilation, caching, parallelism, retries, fallbacks, rate limits, redaction, isolation, and observability without changing semantics silently. Accelerators execute dense and sparse tensor kernels while HBM stores weights, activations, adapters, and KV state; CPUs tokenize and orchestrate; host memory, storage, PCIe, scale-up fabric, and scale-out networks move artifacts and requests. Batch, sequence length, vocabulary, precision, cache locality, communication, and power determine delivered rather than peak behavior. Typical failures include data leakage, template mismatch, tokenizer drift, train-serving skew, stale caches, unsupported operators, precision loss, memory fragmentation, prompt injection, malformed structured output, tool side effects, runaway loops, evaluation contamination, hidden retries, and average metrics that conceal catastrophic tails. A fluent answer is not evidence of correctness.

Evaluation, security, and lifecycle controls. Use multilingual, emoji, code, whitespace, combining marks, invalid bytes, special strings, long inputs, round trips, known ID fixtures, template counts, streaming boundaries, and checkpoint compatibility. Vocabulary size, tokens per byte/word by domain, unknown/fallback rate, sequence and truncation distributions, encode/decode speed, embedding parameters, KV bytes, task quality, latency, and cost matter. Tokens can expose sensitive text in logs and billing; minimize retained raw text/IDs, control special-token authority, document tokenizer language disparities, and audit pricing or quota decisions that affect users unevenly. Verification combines unit and property tests, reference parity, adversarial and edge-case prompts, schema validation, deterministic replay, offline benchmark suites, human review, safety red teaming, privacy and security tests, load and fault injection, long-context checks, shadow traffic, canary rollout, and rollback drills. Every result links to the exact model, data, tokenizer, configuration, code, and runtime. Collection, filtering, training or tuning, evaluation, registration, deployment, monitoring, incident response, refresh, rollback, retention, deletion, and retirement form one lifecycle. Model cards, data and prompt lineage, approvals, exceptions, dependencies, licenses, checkpoints, adapter versions, tool permissions, and evaluation evidence remain auditable. Owners define intended and prohibited use, access and tenant isolation, data minimization, consent or lawful basis, secret handling, human confirmation for consequential actions, rate and spend limits, abuse monitoring, appeal and escalation, retention, and incident responsibility. External model or framework behavior is treated as an untrusted dependency with pinned versions and compensating controls.

<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 760 470" width="100%" height="100%" style="background:#0F172A;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Helvetica,Arial,sans-serif;">
  <defs>
    <linearGradient id="cardGrad" x1="0%" y1="0%" x2="0%" y2="100%">
      <stop offset="0%" stop-color="#1E293B"/>
      <stop offset="100%" stop-color="#0F172A"/>
    </linearGradient>
  </defs>

  <!-- Title Banner -->
  <rect x="20" y="20" width="720" height="56" rx="14" fill="url(#cardGrad)" stroke="#334155" stroke-width="1"/>
  <text x="36" y="46" font-size="18" font-weight="800" fill="#F8FAFC">Tokenization &amp; Subword Encoding Pipeline</text>
  <text x="36" y="64" font-size="12" font-weight="600" fill="#94A3B8">Byte-Pair Encoding (BPE) · WordPiece · SentencePiece · Vocabulary Lookup Tables</text>

  <!-- Panel 1: Text to Token IDs Conversion -->
  <rect x="20" y="90" width="350" height="350" rx="16" fill="url(#cardGrad)" stroke="#334155" stroke-width="1"/>
  <text x="40" y="118" font-size="14" font-weight="700" fill="#38BDF8">1. Text Preprocessing &amp; Subword Split</text>

  <!-- String Input -->
  <rect x="40" y="140" width="310" height="42" rx="8" fill="#0F172A" stroke="#475569" stroke-width="1"/>
  <text x="55" y="160" font-size="11" font-weight="700" fill="#CBD5E1">Raw Unicode String Input:</text>
  <text x="55" y="174" font-size="12" font-weight="800" fill="#38BDF8">"Unbelievable pre-training data"</text>

  <path d="M 195,182 L 195,202" stroke="#94A3B8" stroke-width="2"/>

  <!-- Subword Tokens -->
  <rect x="40" y="202" width="310" height="55" rx="8" fill="#1E293B" stroke="#0284C7" stroke-width="1.5"/>
  <text x="55" y="220" font-size="11" font-weight="700" fill="#CBD5E1">BPE Subword Chunks (Tokenizer):</text>
  <text x="55" y="242" font-size="11" font-weight="800" fill="#A78BFA">["Un", "believ", "able", " pre", "-", "train", "ing"]</text>

  <path d="M 195,257 L 195,277" stroke="#94A3B8" stroke-width="2"/>

  <!-- Vocab ID Array -->
  <rect x="40" y="277" width="310" height="145" rx="10" fill="#0F172A" stroke="#334155" stroke-width="1"/>
  <text x="55" y="298" font-size="12" font-weight="800" fill="#F8FAFC">Numerical Token IDs (Vocabulary Map):</text>
  <text x="55" y="320" font-size="11" font-family="monospace" fill="#38BDF8">[ 3481,  18923,  1204,   742,  14,  3102,  284 ]</text>
  
  <text x="55" y="348" font-size="11" font-weight="700" fill="#F59E0B">Vocab Sizes &amp; Encoding Algorithms:</text>
  <text x="55" y="368" font-size="10" font-weight="600" fill="#CBD5E1">• GPT-4 / tiktoken: ~100k vocabulary size (o200k = 200k)</text>
  <text x="55" y="386" font-size="10" font-weight="600" fill="#CBD5E1">• Llama-3 BPE: 128,000 subword tokens</text>
  <text x="55" y="404" font-size="10" font-weight="600" fill="#CBD5E1">• Special Tokens: &lt;|endoftext|&gt;, &lt;|im_start|&gt;, &lt;|pad|&gt;</text>

  <!-- Panel 2: Tokenization Mechanics -->
  <rect x="390" y="90" width="350" height="350" rx="16" fill="url(#cardGrad)" stroke="#334155" stroke-width="1"/>
  <text x="410" y="118" font-size="14" font-weight="700" fill="#38BDF8">2. BPE Merge Rules &amp; Efficiency</text>

  <!-- BPE Training Algorithm Diagram -->
  <rect x="410" y="135" width="310" height="180" rx="10" fill="#0F172A" stroke="#334155" stroke-width="1"/>
  <text x="425" y="158" font-size="12" font-weight="800" fill="#38BDF8">BPE Pair Merging Iterations:</text>
  
  <text x="425" y="182" font-size="11" font-weight="600" fill="#CBD5E1">Step 1: Base Character Vocab → ['e', 'r', 's', 't']</text>
  <text x="425" y="204" font-size="11" font-weight="600" fill="#CBD5E1">Step 2: Most frequent pair ('e' + 'r') → Merge to 'er'</text>
  <text x="425" y="226" font-size="11" font-weight="600" fill="#CBD5E1">Step 3: Next frequent pair ('s' + 't') → Merge to 'st'</text>
  <text x="425" y="248" font-size="11" font-weight="600" fill="#A78BFA">Step N: Frequent word ('t' + 'est') → Merge to 'test'</text>
  <line x1="425" y1="260" x2="705" y2="260" stroke="#334155" stroke-width="1"/>
  <text x="425" y="280" font-size="10" font-weight="700" fill="#F59E0B">Result: Handles Out-Of-Vocabulary (OOV) gracefully!</text>

  <!-- Compression Metric Box -->
  <rect x="410" y="325" width="310" height="100" rx="10" fill="#1E293B" stroke="#475569" stroke-width="1"/>
  <text x="425" y="348" font-size="12" font-weight="800" fill="#38BDF8">Compression Efficiency:</text>
  <text x="425" y="370" font-size="11" font-weight="600" fill="#F8FAFC">English Ratio: ~1 token ≈ 4 characters / 0.75 words</text>
  <text x="425" y="392" font-size="11" font-weight="600" fill="#CBD5E1">Code / Math: Higher token density per character</text>
  <text x="425" y="412" font-size="10" font-weight="700" fill="#F59E0B">Large Vocabs → Lower sequence length, higher GPU throughput</text>
</svg>

Selection and practical application. Use the model-native tokenizer unless retraining the model, compare segmentation on actual languages and code, prefer byte fallback for coverage, and size vocabulary from quality, sequence, and hardware tradeoffs. Language modeling, translation, search, code, chat, speech codes, image/video latent models, and multimodal systems all operate on tokens. Token behavior links normalization, chat template, vocabulary, embeddings, position, context window, attention, KV cache, decoding, streaming, pricing, and evaluation. The useful optimization boundary is the end-to-end application: user interface, model, tokenizer, context builder, cache, adapter, retriever, tools, runtime, accelerator, scheduler, network, policy, monitoring, and human workflow. Improving one component can move the bottleneck or weaken correctness, safety, isolation, and recoverability elsewhere. A production definition states the base model and revision, tokenizer and vocabulary, context and output limits, numerical precision, data provenance, objective, trainable state, inference runtime, tool or retrieval boundary, evaluation population, latency and cost target, failure policy, and reproducibility artifacts. Similar labels can hide materially different implementations, so exact interfaces and assumptions belong in the contract. Evaluation keeps task quality beside factuality, calibration, robustness, safety, subgroup behavior, context utilization, throughput, time to first token, inter-token latency, tail latency, memory, bandwidth, accelerator utilization, energy, and cost. Controlled comparisons hold prompts, sampling, data, model, hardware, concurrency, and judge protocol fixed and report uncertainty across repeated runs. CFS connects this topic to semiconductor architecture, implementation, verification, manufacturing, packaging, test, and deployed AI-system tradeoffs across the platform.