inference cost

Inference cost is the computational expense of generating outputs from a trained model during deployment, often exceeding training cost over the model's lifetime and driving major architectural and optimization decisions. Cost components: (1) Compute—GPU/TPU time for forward pass (matrix multiplications, attention); (2) Memory—GPU memory for model weights, KV cache, activations; (3) Energy—power consumption per query; (4) Infrastructure—servers, networking, cooling, datacenter. Cost metrics: (1) Cost per token—typically $0.001-0.06 per 1K tokens depending on model size; (2) Cost per query—varies by output length, $0.01-0.50+ for complex queries; (3) Tokens per second per GPU—throughput efficiency; (4) Dollars per GPU-hour—$1-4 for cloud GPU instances. Cost drivers by model size: (1) 7B parameters—~14GB in FP16, runs on single GPU, low cost; (2) 70B—~140GB, requires multi-GPU, 10× cost of 7B; (3) 400B+—requires multi-node, 50-100× cost of 7B. Optimization strategies: (1) Quantization—INT8/INT4 reduces memory and compute 2-4×; (2) KV cache optimization—PagedAttention, multi-query attention reduce memory; (3) Speculative decoding—use small draft model to speed autoregressive generation; (4) Batching—amortize compute across concurrent requests; (5) Pruning/distillation—smaller models with similar quality; (6) Mixture of experts—activate subset of parameters per token. Inference vs. training cost: training is one-time (millions of dollars for frontier models); inference accumulates (can exceed training cost within months for popular services). Hardware trends: inference-optimized chips (Groq, AWS Inferentia, Google TPU v5e) designed for throughput and cost efficiency. Inference cost is the dominant factor in AI economics—driving the entire optimization stack from model architecture to serving infrastructure.