Model orchestration and routing | ChipFoundryServices

Home› Knowledge Base› Model orchestration and routing

Model orchestration and routing is the technique of directing requests to different AI models based on query characteristics — using intelligent routing to send simple queries to fast/cheap models and complex queries to powerful/expensive models, optimizing cost, latency, and quality across a portfolio of AI capabilities.

What Is Model Routing?

Definition: Dynamically selecting which model handles each request.
Goal: Optimize cost, latency, and quality simultaneously.
Methods: Rule-based, classifier-based, or LLM-based routing.
Context: Multiple models with different cost/capability trade-offs.

Why Routing Matters

Cost Optimization: Use expensive models only when needed (90%+ spend reduction possible).
Latency: Fast models for simple queries, powerful for complex.
Quality: Match model capability to task requirements.
Reliability: Fallback to alternate models on failures.
Scalability: Distribute load across model portfolio.

Router Architectures

Rule-Based Routing:

def route(query):
    if len(query) < 50 and "?" not in query:
        return "gpt-3.5-turbo"  # Simple, cheap
    elif "code" in query.lower():
        return "claude-3-sonnet"  # Good at code
    else:
        return "gpt-4o"  # Default capable

Classifier-Based Routing:

Train classifier on:
- Query difficulty labels
- Query category labels
- Historical model performance

At inference:
Query → Classifier → Predicted best model

LLM-Based Routing:

Use small, fast LLM to analyze query:
"Based on this query, which model should handle it?"
→ Route to recommended model

Cascading Strategy

<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
  <rect x="0" y="0" width="760" height="470" fill="#0d1117"/>
  <text x="380" y="28" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">LLM Router — Intelligent Model Selection</text>
  <text x="380" y="48" fill="#8b98a5" font-size="12" text-anchor="middle">classify the query, route to the cheapest model that can handle it — maximize quality per dollar</text>

  <!-- === TOP: Router architecture === -->
  <rect x="25" y="62" width="710" height="140" rx="6" fill="#080d14" stroke="#233043" stroke-width="1.2"/>
  <text x="380" y="82" fill="#e6edf3" font-size="11" text-anchor="middle" font-weight="600">Router Architecture — Query → Classify → Dispatch</text>

  <!-- User query -->
  <rect x="40" y="100" width="80" height="35" rx="4" fill="#14261f" stroke="#34d399" stroke-width="0.9"/>
  <text x="80" y="118" fill="#6ee7b7" font-size="8.5" text-anchor="middle">User query</text>
  <text x="80" y="130" fill="#6b7684" font-size="7" text-anchor="middle">"explain X..."</text>

  <!-- Arrow to router -->
  <path d="M123,117 L160,117" fill="none" stroke="#8b98a5" stroke-width="0.8"/>
  <polygon points="158,114 164,117 158,120" fill="#8b98a5"/>

  <!-- Router classifier -->
  <rect x="168" y="92" width="120" height="52" rx="5" fill="#2a1a0a" stroke="#f59e0b" stroke-width="1.2"/>
  <text x="228" y="112" fill="#fbbf24" font-size="9.5" text-anchor="middle" font-weight="600">Router</text>
  <text x="228" y="126" fill="#a1701a" font-size="8" text-anchor="middle">difficulty classifier</text>
  <text x="228" y="138" fill="#6b7684" font-size="7" text-anchor="middle">or embedding similarity</text>

  <!-- Routing arrows to 3 model tiers -->
  <path d="M291,100 L345,90" fill="none" stroke="#34d399" stroke-width="0.9"/>
  <path d="M291,117 L345,117" fill="none" stroke="#60a5fa" stroke-width="0.9"/>
  <path d="M291,134 L345,145" fill="none" stroke="#c4b5fd" stroke-width="0.9"/>

  <!-- Small model -->
  <rect x="348" y="78" width="130" height="28" rx="4" fill="#14261f" stroke="#34d399" stroke-width="0.9"/>
  <text x="413" y="96" fill="#6ee7b7" font-size="8.5" text-anchor="middle">Small (GPT-4o-mini, Haiku)</text>

  <!-- Medium model -->
  <rect x="348" y="110" width="130" height="28" rx="4" fill="#0f1a2a" stroke="#60a5fa" stroke-width="0.9"/>
  <text x="413" y="128" fill="#93c5fd" font-size="8.5" text-anchor="middle">Medium (Sonnet, GPT-4o)</text>

  <!-- Large model -->
  <rect x="348" y="142" width="130" height="28" rx="4" fill="#1a1520" stroke="#a78bfa" stroke-width="0.9"/>
  <text x="413" y="160" fill="#c4b5fd" font-size="8.5" text-anchor="middle">Large (Opus, o1-pro)</text>

  <!-- Cost annotations -->
  <text x="500" y="96" fill="#34d399" font-size="8">$0.15/MTok — 80% of queries</text>
  <text x="500" y="128" fill="#60a5fa" font-size="8">$3/MTok — 15% of queries</text>
  <text x="500" y="160" fill="#c4b5fd" font-size="8">$15/MTok — 5% of queries</text>

  <!-- Result back -->
  <text x="380" y="188" fill="#8b98a5" font-size="8.5" text-anchor="middle">Blended cost: ~$0.80/MTok average (vs $3/MTok if always using medium)</text>

  <!-- === MIDDLE LEFT: Routing strategies === -->
  <rect x="25" y="210" width="350" height="115" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="200" y="228" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Routing Strategies</text>

  <text x="45" y="250" fill="#60a5fa" font-size="8.5" font-weight="600">Classifier-based</text>
  <text x="160" y="250" fill="#8b98a5" font-size="8.5">small model predicts difficulty tier</text>
  <text x="45" y="268" fill="#34d399" font-size="8.5" font-weight="600">Embedding similarity</text>
  <text x="185" y="268" fill="#8b98a5" font-size="8.5">match query to capability clusters</text>
  <text x="45" y="286" fill="#fbbf24" font-size="8.5" font-weight="600">Cascading</text>
  <text x="120" y="286" fill="#8b98a5" font-size="8.5">try small first, escalate if uncertain</text>
  <text x="45" y="304" fill="#c4b5fd" font-size="8.5" font-weight="600">Rule-based</text>
  <text x="120" y="304" fill="#8b98a5" font-size="8.5">keyword/regex → model mapping</text>
  <text x="45" y="320" fill="#f87171" font-size="8.5" font-weight="600">LLM-as-router</text>
  <text x="145" y="320" fill="#8b98a5" font-size="8.5">cheap model classifies (self-routing)</text>

  <!-- === MIDDLE RIGHT: What to route on === -->
  <rect x="390" y="210" width="345" height="115" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="562" y="228" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Routing Signals</text>

  <text x="410" y="250" fill="#8b98a5" font-size="8.5">• Task complexity (simple Q&amp;A vs multi-step reasoning)</text>
  <text x="410" y="268" fill="#8b98a5" font-size="8.5">• Required capabilities (code, math, creative, factual)</text>
  <text x="410" y="286" fill="#8b98a5" font-size="8.5">• Context length (short vs 100K+)</text>
  <text x="410" y="304" fill="#8b98a5" font-size="8.5">• Latency requirements (real-time vs batch)</text>
  <text x="410" y="320" fill="#8b98a5" font-size="8.5">• User tier (free vs paid vs enterprise)</text>

  <!-- === BOTTOM: Implementations === -->
  <rect x="25" y="335" width="710" height="68" rx="5" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="380" y="353" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Production Routing Systems</text>

  <text x="105" y="375" fill="#c4b5fd" font-size="9.5" text-anchor="middle" font-weight="600">OpenRouter</text>
  <text x="105" y="389" fill="#8b98a5" font-size="8" text-anchor="middle">multi-provider API routing</text>

  <text x="260" y="375" fill="#fbbf24" font-size="9.5" text-anchor="middle" font-weight="600">Martian</text>
  <text x="260" y="389" fill="#8b98a5" font-size="8" text-anchor="middle">learned router, quality-aware</text>

  <text x="415" y="375" fill="#34d399" font-size="9.5" text-anchor="middle" font-weight="600">Not Diamond</text>
  <text x="415" y="389" fill="#8b98a5" font-size="8" text-anchor="middle">per-query model selection</text>

  <text x="570" y="375" fill="#60a5fa" font-size="9.5" text-anchor="middle" font-weight="600">Unify</text>
  <text x="570" y="389" fill="#8b98a5" font-size="8" text-anchor="middle">benchmark-based routing</text>

  <text x="695" y="375" fill="#f87171" font-size="9.5" text-anchor="middle" font-weight="600">Custom</text>
  <text x="695" y="389" fill="#8b98a5" font-size="8" text-anchor="middle">self-hosted classifiers</text>

  <!-- Key insight -->
  <rect x="25" y="411" width="710" height="22" rx="3" fill="#0b1220" stroke="#233043" stroke-width="0.8"/>
  <text x="380" y="426" fill="#fbbf24" font-size="9" text-anchor="middle">80% of queries are easy enough for the cheapest model — routing is the highest-leverage cost optimization.</text>

  <text x="380" y="460" fill="#6b7684" font-size="11" text-anchor="middle">Model routing turns the explosion of available LLMs from a burden into an advantage — right model, right cost, every time.</text>
</svg>

Multi-Model Portfolios

Model            | Cost/1M tk | Latency | Capability | Use For
-----------------|------------|---------|------------|------------------
GPT-3.5-turbo    | $0.50      | ~200ms  | Basic      | Simple Q&A, chat
GPT-4o-mini      | $0.15      | ~300ms  | Good       | General tasks
GPT-4o           | $5.00      | ~500ms  | Strong     | Complex reasoning
Claude-3.5-Sonnet| $3.00      | ~400ms  | Strong     | Code, writing
Claude-3-Opus    | $15.00     | ~800ms  | Strongest  | Critical tasks
Llama-3.1-8B     | ~$0.05*    | ~100ms  | Basic      | High-volume simple

*Self-hosted estimate

Routing Signals

Query Characteristics:

Length: Short queries → simpler model.
Keywords: Domain-specific → specialized model.
Complexity: Multi-hop reasoning → powerful model.
Format: Code, math, writing → specialized model.

User/Context:

Customer tier: Premium → best model.
History: Past failures → try different model.
SLA: Low latency required → fast model.

System State:

Load: High traffic → distribute to cheaper models.
Errors: Primary down → automatic fallback.
Cost budget: Near limit → prefer cheaper.

Ensemble Strategies

Best-of-N:

1. Send query to N models
2. Collect all responses
3. Use judge model to pick best
4. Return winning response

Expensive but highest quality

Consensus Checking:

1. Send to 2+ models
2. If responses agree → return any
3. If different → escalate to powerful model

Good for factual accuracy

Orchestration Platforms

LiteLLM: Unified API for 100+ model providers.
Portkey: AI gateway with routing, caching, fallbacks.
Martian: Intelligent model router.
OpenRouter: Multi-provider routing.
Custom: Build with simple routing logic.

Implementation Example

class ModelRouter:
    def __init__(self):
        self.classifier = load_classifier(""router_model.pt"")
        self.models = {
            ""simple"": ""gpt-3.5-turbo"",
            ""moderate"": ""gpt-4o-mini"",
            ""complex"": ""gpt-4o""
        }
    
    def route(self, query: str) -> str:
        complexity = self.classifier.predict(query)
        model = self.models[complexity]
        return call_model(model, query)
    
    def cascade(self, query: str) -> str:
        for model in [""simple"", ""moderate"", ""complex""]:
            response, confidence = call_with_confidence(
                self.models[model], query
            )
            if confidence > 0.85:
                return response
        return response  # Final attempt

Model orchestration and routing is essential for production AI economics — without intelligent routing, teams either overspend on powerful models for simple tasks or underserve complex queries with weak models, making routing architecture critical for balancing cost, quality, and user experience.

orchestratorroutermulti-modelroutingmodel selectioncascadeensemblecost optimization

Explore 500+ Semiconductor & AI Topics

From EUV lithography to CUDA optimization — search the full knowledge base or chat with our AI assistant.

🔍 Search Topics 💬 Ask CFSGPT 📚 Browse All