# How to Evaluate an LLM Router for Production

> A source-based guide to evaluating routing decisions, task shifts, prompt-cache effects, observability, credentials, and production rollout.

Source: https://ailoot.ai/blog/en/evaluate-llm-router-production/
Canonical: https://ailoot.ai/blog/en/evaluate-llm-router-production/
Language: en
Publisher: Ailoot (https://ailoot.ai/)
Translation: guide-100017
Revision: sha256-89ef06e207d57837f6795d6f39b86c2a79586f129db955e881c624ddd1a8c61e


## How do you pick one for your own workload?

**1. Log the fields that decide everything:** Per call: model, input tokens, cached input tokens, cache write tokens, output tokens, latency, task type, and whether the result was accepted or retried. The cached-input field is the one teams skip and the one that determines whether routing can help you at all. A week of this costs an afternoon of instrumentation.

**2. Compute your ceiling before you shop:** Maximum savings is S = f x (1 - r), where f is the share of calls that can leave the frontier model and r is the cheap model's cost as a fraction of the frontier model's. Both come out of step one.  Spend the time on prompt caching, batch endpoints and reasoning effort instead.

**3. Build a golden set from your own traffic, per task class:** Fifty to two hundred real requests with known-good outcomes, sampled from your logs rather than written by hand. This is the artifact that makes every later conversation with a vendor concrete. Without it you are comparing marketing.

**4. Test the two conditions that predict production behavior:** First, shadow the router on live traffic and compare routed spend against an all-frontier baseline priced with a warm cache, which is the only fair comparison. Second, and this is the step nobody takes, hold out a task class the router has never seen and measure it there. Section 2 is why. A router that looks fine on your calibration set and collapses on a new feature area will not announce itself.

**5. Gate the rollout on the eval, and keep running it:** Require the cheap model to clear a stated bar on each task class before that class becomes routable. Then keep the job running, because model versions change underneath you and a router calibrated in March is describing a model lineup that no longer exists.

## What separates a production LLM router from a demo?

### 1. Optimality, not the headline saving

A saving figure with no quality figure beside it is half a sentence.

Optimality is the axis nobody markets on, and it is the one that decides your bill. It asks how often the router picked the cheapest model that still answered correctly. A router can post a good accuracy number while burning money.

Ask any vendor for optimal selection ratio. If they cannot produce it, they have not measured the thing you are buying.

### 2. Survival when the task type changes

Buy the router that has seen traffic shaped like yours, and insist on being shown a result on a distribution it was not tuned on.

### 3. What it does when the cache is warm

A router is only as good as the mechanism underneath it.

[Anthropic publishes the multipliers](https://platform.claude.com/docs/en/build-with-claude/prompt-caching): a cache read costs 0.1x base input, a five-minute cache write costs 1.25x, a one-hour write costs 2x. So a switch turn costs roughly twelve times what staying put would have cost on the same prefix, and pays back only if the new model holds the session.

These are reconcilable. The provider keeps your abandoned prefix warm, so coming back is cheap; the expensive part is the write on the model you moved to. Whether that trade pays depends on how long your prefixes are and how often you return. Nobody can answer it for you, which is the point.

### 4. Where the routing boundary sits

A router that re-decides on every turn is making a decision it does not have the information to make, and paying a cache write each time it changes its mind.

Good routing boundaries are the ones a session does not cross back over: a whole sub-task, a distinct phase, a background job.

### 5. Whether you can see and override the decision

When output quality drops, an engineer needs to answer one question fast: did the model change, or did I? A router that logs which model it chose, why, and what it cost is debuggable. One that does not is a source of mystery regressions.

Insist on a per-request model field, a per-request cost field, and an override.

### 6. Blast radius, because the router holds your keys

A router sits on the credential path by construction. It has provider keys, and in a self-hosted deployment it may also have cloud credentials. That makes it a high-value dependency. When you evaluate a router, ask how it is distributed, whether releases are signed, whether the deployment path pins dependencies, and what it can read on the host. Self-hosted is not automatically safer than managed; it is differently exposed.

### 7. Who owns it in eighteen months

It is a reason to ask, in the evaluation call, what the roadmap looks like under current ownership and what the migration path is if it stops.

## When should you not use an LLM router?

**When your traffic is uniformly hard:** Routing savings track how uneven your traffic is, not how clever the router is.

## What to do next from here?

You now have the two questions that separate a router that will work from one that will demo well: what is its optimal selection ratio, and what happens when the task type changes.

Start with step one. Add cached-input tokens and task type to your request logs and leave it running for a week. That single field is the difference between a real value for f and a guess.

## Frequently Asked Questions

### What is an LLM router?

An LLM router is a decision layer between your application and a pool of models that sends each request to the model most likely to complete it acceptably at the lowest cost. Pre-generation routers decide from the prompt alone; cascading routers let a cheap model answer first and escalate when a scorer rejects the answer. Most production systems now combine both.

### Do LLM routers actually save money?

Yes, but far less than headline figures suggest, and the amount depends on your traffic rather than the router. Compute S = f x (1 - r) from your own logs before believing any vendor figure.

### What is the difference between an LLM router and an AI gateway?

A gateway gives you one endpoint, failover, key management, budgets and observability across many providers. A router decides which model handles a given request. Most gateways now ship a router, but the two capabilities are independent.

### Can I build my own LLM router instead of buying one?

You can, and the measurement work is identical either way, so do steps one to three of the sequence above first. If you build, start with a per-task-type lookup table. Add a learned classifier only when that baseline demonstrably falls short.

### Does switching models break prompt caching?

Switching away costs you a cache write on the new model, billed at 1.25x base input for a five-minute TTL against 0.1x for the read you abandoned. The trade depends on prefix length and how often you return, so measure it on your own traffic.

## Put this guide to work with Ailoot

Connect AI models to your applications, agents and automations through one Ailoot API. Choose the model that fits your task and follow the [Ailoot documentation](https://ailoot.ai/docs/) to get started.


## Ailoot · The company and its project portfolio

Ailoot combines established operating experience, a large client base, stable service and ongoing KPI control.

The company and its project portfolio: millions of customers; monthly sales in the billions.

Ailoot brings that operating experience to API access for AI features in customer products, agents, applications and automations.
