tokens per second

**Tokens per second** is the **throughput metric for language-model workloads representing text units processed or generated per second** - it is the standard speed measure for both LLM training and inference economics. **What Is Tokens per second?** - **Definition**: Count of tokenizer output units handled each second during training or generation. - **Training vs Inference**: Training tracks processed tokens per step, while inference tracks generated-token rate. - **Determinants**: Model size, context length, batching strategy, and hardware communication efficiency. - **Economic Link**: Higher token throughput improves tokens-per-dollar and reduces time-to-result. **Why Tokens per second Matters** - **LLM Performance Baseline**: Provides common cross-model and cross-cluster speed benchmark. - **Cost Management**: Token throughput strongly influences cloud spend for training and serving. - **Capacity Forecasting**: Supports planning for latency and throughput service-level targets. - **Optimization Visibility**: Reveals impact of kernel, quantization, and parallelism improvements. - **Product Relevance**: Inference token rate directly affects user-perceived responsiveness. **How It Is Used in Practice** - **Measurement Consistency**: Standardize tokenizer, sequence length, and warm-up treatment when reporting. - **System Correlation**: Analyze token rate alongside GPU utilization and memory bandwidth metrics. - **Policy Tuning**: Adjust batching, sequence packing, and model parallel strategy for better token throughput. Tokens per second is **the operational speed currency of language-model systems** - precise tracking connects engineering tuning directly to user experience and cost efficiency.