portkey
**Portkey** is a **production-grade AI Gateway and LLMOps platform that provides reliability, cost optimization, and full observability for LLM applications** — acting as a smart reverse proxy between your application and AI providers, with automatic fallbacks, semantic caching, detailed tracing, and budget controls that transform LLM API calls from fragile one-off requests into managed, monitored infrastructure.
**What Is Portkey?**
- **Definition**: A managed AI Gateway (cloud-hosted or self-hosted) and observability platform that intercepts LLM API calls through an OpenAI-compatible endpoint — adding reliability features (fallbacks, retries, load balancing), cost optimization (semantic caching, budget limits), and full observability (tracing, cost tracking, user analytics) transparently.
- **Gateway Model**: Applications send requests to Portkey's OpenAI-compatible endpoint instead of directly to providers — a single line change enables all Portkey features without modifying application logic.
- **Provider Coverage**: Routes to 200+ AI providers and models — OpenAI, Anthropic, Azure, Google Vertex, AWS Bedrock, Together AI, Groq, Ollama, and any OpenAI-compatible endpoint.
- **Config-Based Routing**: Routing logic (fallbacks, load balancing, caching) is defined in JSON configs stored in Portkey — decoupled from application code and changeable without redeployment.
- **Enterprise Focus**: Designed for teams managing LLM spend at scale — per-user budgets, team-level access controls, audit logs, and SSO integration.
**Why Portkey Matters**
- **Reliability at Scale**: Single provider outages don't bring down your application — Portkey automatically routes to fallback providers with sub-second switchover, maintaining user experience during OpenAI or Anthropic incidents.
- **Cost Reduction**: Semantic caching (not just exact match) can reduce API costs by 20-40% for applications with similar repeated queries — a user asking "What's the weather?" and another asking "Tell me the weather" can share a cached response.
- **Unified Observability**: Every request — across all providers, all models, all users — appears in a single dashboard with latency, cost, token usage, and error rate — replacing scattered per-provider monitoring.
- **Prompt Management**: Store, version, and A/B test prompts in Portkey's prompt library — deploy prompt changes without code releases.
- **Multi-Tenant Control**: Route different users or teams to different models, apply different rate limits, and track costs per customer — essential for SaaS products billing customers for AI usage.
**Core Portkey Features**
**Automatic Fallbacks**:
```python
import portkey_ai
portkey = portkey_ai.Portkey(api_key="pk-...", config={
"strategy": {"mode": "fallback"},
"targets": [
{"provider": "openai", "api_key": "sk-..."},
{"provider": "anthropic", "api_key": "sk-ant-..."}
]
})
# If OpenAI fails, automatically retries on Anthropic — transparent to caller
response = portkey.chat.completions.create(model="gpt-4o", messages=[...])
```
**Load Balancing**:
```python
config = {
"strategy": {"mode": "loadbalance"},
"targets": [
{"provider": "openai", "weight": 0.7}, # 70% of traffic
{"provider": "azure-openai", "weight": 0.3} # 30% of traffic
]
}
```
**Semantic Caching**:
```python
portkey = portkey_ai.Portkey(api_key="pk-...", cache={"mode": "semantic", "max_age": 3600})
# Requests semantically similar to cached queries return cached results — no LLM call
```
**Observability Features**
- **Request Tracing**: Every LLM call recorded with input, output, latency, tokens, cost, model, provider, and user ID.
- **Cost Analytics**: Daily/weekly/monthly spend by model, provider, user, or custom metadata tag — budget forecasting and anomaly detection.
- **Error Analysis**: Automatic categorization of errors (rate limits, context length, content policy) with retry rates and failure patterns.
- **Feedback Integration**: Attach user feedback (thumbs up/down, CSAT scores) to traces for quality monitoring.
- **Custom Metadata**: Tag requests with `user_id`, `session_id`, `feature_name` — filter any metric by any dimension.
**Portkey vs Competitors**
| Feature | Portkey | LiteLLM Proxy | Helicone | Direct API |
|---------|---------|--------------|---------|-----------|
| Semantic caching | Yes | No | Yes | No |
| Fallbacks | Yes | Yes | No | Manual |
| Observability | Comprehensive | Basic | Good | None |
| Prompt management | Yes | No | No | Manual |
| Self-hostable | Yes (Enterprise) | Yes | Yes | N/A |
| Provider count | 200+ | 100+ | 50+ | 1 |
**Deployment Modes**
- **Cloud Gateway**: Use Portkey's managed endpoint — zero infrastructure, instant setup, 99.99% uptime SLA.
- **Self-Hosted**: Deploy Portkey Gateway on your own infrastructure — data never leaves your environment, required for regulated industries (healthcare, finance).
- **SDK Integration**: Python and TypeScript SDKs for programmatic config management and metadata attachment.
Portkey is **the production LLM infrastructure layer that transforms unreliable AI API calls into managed, observable, cost-optimized services** — for teams moving from prototype to production with LLM applications, Portkey provides the reliability and visibility that enterprise applications require without the months of custom infrastructure development.