cost
**LLM pricing and costs** are the **economic factors that determine the total expense of running AI applications** — including API costs per token, self-hosting infrastructure expenses, and optimization strategies, critical for building sustainable AI products and making build-vs-buy decisions.
**What Are LLM Costs?**
- **Definition**: Total expense of using LLMs in production.
- **Components**: API fees, infrastructure, optimization, engineering.
- **Unit**: Typically cost per million tokens (input and output separately).
- **Variation**: 100× difference between cheapest and most expensive options.
**Why Pricing Matters**
- **Product Economics**: AI features must be profitable.
- **Build vs. Buy**: Self-hosting vs. API decision.
- **Architecture Choices**: Model routing, caching, batching decisions.
- **Scale Planning**: Costs compound at scale.
- **Competitive Position**: Lower costs enable lower prices or higher margins.
**API Pricing Comparison (2024)**
```
Provider/Model | Input/1M tk | Output/1M tk | Notes
------------------------|-------------|--------------|---------------
GPT-4o | $2.50 | $10.00 | Most capable
GPT-4o-mini | $0.15 | $0.60 | Cost-optimized
GPT-3.5-turbo | $0.50 | $1.50 | Legacy
Claude 3.5 Sonnet | $3.00 | $15.00 | Strong reasoning
Claude 3 Haiku | $0.25 | $1.25 | Fast, cheap
Gemini 1.5 Pro | $1.25 | $5.00 | Long context
Gemini 1.5 Flash | $0.075 | $0.30 | Fastest
Llama 3.1 70B (hosted) | $0.20-0.80 | $0.20-0.80 | Varies by host
Mistral Large | $2.00 | $6.00 | European option
```
**Self-Hosting Economics**
**Infrastructure Costs**:
```
Hardware Option | Monthly Cost | Models Served
-------------------|--------------|--------------------
RTX 4090 (24GB) | ~$500 amort. | 7-13B models
A100 40GB | $2-3K cloud | Up to 30B
A100 80GB | $3-4K cloud | Up to 70B
H100 80GB | $4-6K cloud | 70B+ fast inference
8× H100 cluster | $30-40K | Any model, high throughput
```
**Break-Even Analysis**:
```
API cost example: $5/M tokens × 10M tokens/day = $50/day = $1,500/month
H100 cost: ~$5,000/month
Break-even: ~100M tokens/day for H100
Below this: API often cheaper
Above this: Self-host saves money
```
**Cost Optimization Strategies**
**Caching**:
```
Common queries → Cache responses
Hit rate of 20% → 20% cost reduction
Semantic caching: Similar queries hit cache
Implement: Redis, custom cache layer
```
**Model Routing**:
```
Simple queries → Cheap/small model (90% of traffic)
Complex queries → Expensive/large model (10% of traffic)
Potential savings: 60-80%
```
**Prompt Optimization**:
```
Before: 2,000 token system prompt
After: 500 token optimized prompt
Savings: 75% on input tokens
Techniques:
- Compression
- Remove redundancy
- Batch instructions
```
**Output Control**:
```
max_tokens: Set appropriate limits
Stop sequences: End early when possible
JSON mode: Structured output (often shorter)
```
**Batching**:
```
Real-time: Process individually (higher per-request cost)
Batch: Accumulate, process together (lower per-request cost)
When acceptable latency allows, batch for savings
```
**Cost Tracking**
**What to Measure**:
- Tokens per request (input + output).
- Requests per user/feature.
- Cost per user action.
- Cost per successful outcome.
**Implementation**:
```python
class CostTracker:
def __init__(self):
self.costs = defaultdict(float)
def record(self, user_id, feature,
input_tokens, output_tokens, model):
cost = calculate_cost(
input_tokens, output_tokens, model
)
self.costs[user_id] += cost
self.costs[feature] += cost
self.log(user_id, feature, cost)
```
**Cost by Use Case**
```
Use Case | Typical Cost | Optimization
----------------------|-------------------|-------------------
Chat (1 turn) | $0.001-0.01 | Cache, small model
Code completion | $0.0001-0.001 | Small model, prefix caching
Document summary | $0.01-0.10 | Batch, smaller model
RAG (search + answer) | $0.005-0.05 | Cache embeddings
Agent (multi-step) | $0.10-1.00 | Limit retries, cheaper tools
```
**Cost Control Architecture**
```svg
```
LLM pricing and costs are **the foundation of AI product economics** — understanding and optimizing costs determines whether AI features are sustainable at scale, making cost engineering as important as prompt engineering for production AI systems.