AI Factory Glossary

token dropping, llm architecture

Token dropping discards excess tokens when expert capacity is exceeded.

token dropping,optimization

Skip processing some tokens to save compute.

token forcing, llm optimization

Token forcing mandates specific tokens at designated positions.

token healing, text generation

Fix tokenization artifacts.

token importance scoring, llm architecture

Assign importance scores to determine computation allocation.

token labeling in vit, computer vision

Assign labels to patches.

token labeling, computer vision

Assign pseudo-labels to tokens.

token limit in prompts, generative models

Maximum prompt length.

token merging rules, nlp

How to combine tokens.

token merging, transformer

Combine similar tokens to reduce sequence length.

token pruning, optimization

Remove unimportant tokens.

token streaming, llm optimization

Token streaming sends individual tokens immediately rather than waiting for completion.

token tree search, inference

Search over possible continuations.

token-to-parameter ratio, training

Training tokens per parameter.

token,tokenize,tokenization,bpe

Token: Subword unit. GPT uses BPE tokenization. ~1 token = 4 characters. Models predict probability of next token.

tokenization artifacts, challenges

Problems from subword tokenization.

tokenization consistency, nlp

Ensure consistent tokenization.

tokenization normalization, nlp

Preprocessing before tokenization.

tokenization security,security

Protect tokens from injection or manipulation.

tokenization,bpe,sentencepiece

Tokenization splits text into tokens. BPE, SentencePiece, tiktoken. Vocabulary size trade-off (32K-100K typical).

tokenization,nlp

Split text into tokens (subwords characters) for model input.

tokenizer training, nlp

Create tokenizer vocabulary.

tokenizer,bpe,wordpiece,spm

Tokenizer splits text into tokens. BPE/WordPiece/SentencePiece trade off vocabulary size vs. sequence length and handle multilingual text differently.

tokenizers,fast,rust

Tokenizers library provides fast tokenization. Rust implementation. Hugging Face.

tokens per second, optimization

Language model training speed.

tolerance design, quality & reliability

Tolerance design balances specification tightness with cost and capability.

tolerance, spc

Allowed variation from target.

tombstoning, quality

Component standing on end.

tool availability,production

Percentage of time tool is ready.

tool calling agent, ai agents

Tool calling agents invoke external functions APIs or resources to accomplish tasks.

tool calling with validation,ai agent

Validate tool call arguments before execution.

tool capacity planning, production

Ensure sufficient equipment.

tool contamination, contamination

Contamination from equipment.

tool discovery, ai agents

Tool discovery enables agents to find and learn about available functions dynamically.

tool documentation, ai agents

Tool documentation describes function capabilities parameters and expected outputs for agent understanding.

tool idle management, environmental & sustainability

Tool idle management powers down unused equipment components reducing standby energy consumption.

tool matching maintenance, production

Keep tools similar.

tool performance specifications, production

Detailed capability requirements.

tool qualification after pm, production

Verify tool performance post-maintenance.

tool qualification, production

Validate equipment meets requirements.

tool result parsing, ai agents

Tool result parsing extracts relevant information from function outputs for agent reasoning.

tool selection, ai agents

Tool selection chooses appropriate functions from available repertoire for current needs.

tool selection, tool use

Choose appropriate tool.

tool sequencing, tool use

Chain multiple tools.

tool suite,production

Group of tools performing same process step.

tool use / function use,ai agent

LLM decides when and how to call external APIs tools or functions.

tool use evaluation, evaluation

Assess tool-using capabilities.

tool use training, fine-tuning

Teach models to use external tools.

tool use, prompting techniques

Tool use enables language models to invoke external functions APIs or search engines.

tool utilization optimization, production

Maximize productive time.