# What Is Model Routing?

> Defines model routing and outlines criteria for evaluating routing rules, aliases, provider distribution, retry and fallback behavior, load balancing, and cost/latency trade-offs.

Source: https://ailoot.ai/blog/en/what-is-model-routing/
Canonical: https://ailoot.ai/blog/en/what-is-model-routing/
Language: en
Publisher: Ailoot (https://ailoot.ai/)
Translation: guide-100045
Revision: sha256-c9be25bc096a975605fd8135626417111af9ece0eb99ded76514d9890de7a26a


Model routing is the process of mapping an incoming LLM request to a specific provider, model identifier, and API key at request time, according to policy rather than application code. A model router evaluates request context (headers, tenant, budget state, prompt complexity) and provider state (error rate, latency, rate-limit headroom), then rewrites the target before the provider call. A model routing layer that cuts LLM token costs is usually where teams first see the return.

The term covers three mechanisms that are often conflated:

- **Static routing** maps a model name to providers with fixed weights.
- **Conditional routing** evaluates an expression per request (tier header, team, budget consumed, complexity tier) and overrides the target on a match.
- **Adaptive routing** re-weights providers and keys continuously from observed error rates and latency.

A production LLM router needs all three; a tool offering one pushes the other two back into application code. The smart LLM routing guide on picking the optimal model per request covers the second mechanism in depth.

## How to Evaluate an LLM Router

| Criterion | What to check |
|----|----|
| Routing rule language | Can a rule read headers, tenant, budget usage, and request type, with defined precedence? |
| Model aliasing | Can `best-model` resolve to different identifiers per key, team, or environment with no code change? |
| Weighted distribution | Are weights applied at provider and API key level, normalized per model? |
| Fallback and retry semantics | Does each fallback get its own retry budget? Is `429` handled differently from `401`? |
| Adaptive load balancing | Are weights recomputed from live error rate and latency, and how fast does a recovered route return? |
| Cost awareness | Is pricing data synced and overridable, and can a rule react to budget consumed? |
| Overhead and deployment | Measured latency at sustained throughput; self-hosted, hosted, or library only? |

The retry versus fallback distinction is the criterion most often missing from vendor comparisons: a retry re-attempts the same provider, a fallback moves to a different one, and a router that merges them cannot budget them separately.

## Common Challenges with Model Routing Tools

Most production model routing failures trace back to a few recurring gaps, each with a configuration-level answer.

- **Routing state lives in application code.** Model names, provider preferences, and fallback lists embedded in services cannot change without a deploy; virtual key configuration and routing rules make them auditable.
- **Retries that ignore error class.** Retrying a `401` with backoff wastes seconds and never succeeds; a router should mark the credential dead and rotate immediately..
- **No record of the routing decision.** Provider rate-limit behavior, such as OpenAI's tiered RPM and TPM limits, is far easier to reason about when the gateway logs which engine chose the route and which key hit a limit.

## Frequently Asked Questions

### What is an LLM router?

An LLM router is a component that selects the provider, model, and credential for each request based on configured rules, static weights, or live performance metrics. It sits between the application and the providers, rewrites the request target before the call, and handles retries and fallbacks when the route fails.

### What is the difference between a model router and an AI gateway?

A model router decides where a request goes. An AI gateway executes the request and also owns credentials, budgets, rate limits, guardrails, and logging.

### How does model routing reduce LLM costs?

Model routing reduces cost by sending each request to the least expensive model that meets its quality requirement and by shifting traffic away from providers close to a budget or rate limit.

### What is the difference between retries and fallbacks?

A retry re-attempts the same provider, optionally with a different API key, after a transient failure such as a `5xx` or `429`. A fallback moves to a different provider and model after the primary has exhausted its retries.

### What is adaptive load balancing in an LLM router?

Adaptive load balancing recomputes route weights from observed metrics instead of static configuration.

## Put this guide to work with Ailoot

Connect AI models to your applications, agents and automations through one Ailoot API. Choose the model that fits your task and follow the [Ailoot documentation](https://ailoot.ai/docs/) to get started.


## Ailoot · The company and its project portfolio

Ailoot combines established operating experience, a large client base, stable service and ongoing KPI control.

The company and its project portfolio: millions of customers; monthly sales in the billions.

Ailoot brings that operating experience to API access for AI features in customer products, agents, applications and automations.
