cost per token

**Cost per token** is the standard pricing metric for **LLM inference services**, measuring how much it costs to process or generate a single **token** (roughly ¾ of a word in English). It is the fundamental unit of economics for deploying and using large language models at scale. **Typical Pricing Structure** - **Input Tokens**: Charged at a lower rate — these are the tokens in your prompt that the model reads and processes. - **Output Tokens**: Charged at a higher rate (often **2–4× input cost**) — these are the tokens the model generates in its response. - **Example**: OpenAI's GPT-4o charges approximately **$2.50 per 1M input tokens** and **$10 per 1M output tokens** (as of early 2025). **What Drives Cost Per Token** - **Model Size**: Larger models with more parameters require more GPU compute per token, directly increasing cost. - **Hardware**: The type of GPU or accelerator (**H100, A100, TPU v5**) and its utilization rate heavily influence unit economics. - **Batch Size**: Higher batch sizes improve GPU utilization and reduce cost per token through better **throughput**. - **Quantization**: Running models in **INT8, INT4, or FP8** precision reduces memory and compute requirements, lowering cost. - **Infrastructure**: Self-hosted vs. cloud API pricing reflects different overhead structures. **Why It Matters** At scale, cost per token determines whether an AI application is **economically viable**. A chatbot handling millions of conversations per day can spend **thousands of dollars per hour** on inference. Optimizing cost per token through model selection, quantization, caching, and efficient batching is a critical engineering challenge.