Sign in
auto router

Start Shadow Eval

Start a shadow eval: duplicate a sampled slice of one or more targets' live traffic against a second arm, judge the two responses blind, and stratify win rates by tier, by the model that served the real arm, and by target.

A target is a virtual key, a team, or a user. Team and user targets match on the identity every request resolves to at auth time, so they cover JWT-authenticated traffic, which presents no virtual key; a user target samples that user's traffic across all their teams, whether it arrives on a JWT or a key they own. models narrows every target to requests for those model groups, so a user plus one model samples that user's traffic on that model across every key they own; it is forward-only, since a reverse job already samples exactly the traffic its own router served.

A forward job answers whether the targets should adopt router_name: it samples the requests the router did not serve and duplicates them through it. A reverse job answers whether a target already on the router still gains from it: it samples the requests the router did serve and duplicates them against baseline_model. A target can hold one active job per direction, so both questions can run at once, and a request matching several jobs' targets (say its key and its team) is sampled by each, separately budgeted.

Shadow responses are never served to users. Each target samples until its recorded eval spend, the shadow and judge calls' own cost, reaches max_budget dollars, the job's window ends, or the job is stopped, so one target running out of budget does not end sampling for the others; sampling changes propagate to pods within about 10 seconds. Shadow and judge calls bill to the sampled request's own identity but are excluded from request counts and auto-router adoption metrics.

post/auto_router/shadow_eval/start

Request body

api_key_idsstring[]

Hashed virtual keys whose traffic will be shadowed. Combined with team_ids and user_ids the job needs at least one target and at most 100, which also bounds every read the job's endpoints make. Each target carries its own max_budget spend budget, so one exhausting its budget leaves the others sampling.

team_idsstring[]

Teams whose traffic will be shadowed, matched on the team every authenticated request resolves to, so a team's JWT-auth and virtual-key traffic are both sampled

user_idsstring[]

Users whose traffic will be shadowed, matched on the user every authenticated request resolves to across all their teams: JWT requests carrying their subject claim and virtual keys they own

modelsstring[]

Model groups to narrow the sampled traffic to, matched on the group the caller requested and resolved through model_group_alias, so an alias and its target are one name. Empty samples every model the targets use. This ANDs with the targets: a job over a user and one model samples that user's requests on that model across every key they own, and none of their other traffic. Forward jobs only: a reverse job samples exactly the traffic its own router served, which no other model group can name

router_namestring nullable

The auto-router under evaluation, in either direction: the single-router spelling of router_names. Provide exactly one of the two fields

router_namesstring[]

The auto-routers under evaluation, at most 4. Every sampled request runs through every router listed and each arm is judged independently against the same real response, so routers compare head-to-head on identical traffic. More than one router requires direction 'forward'. After validation this field always carries the full deduplicated set, whichever spelling the caller used

direction'forward' | 'reverse'

forward answers 'should this key adopt router_name': it samples the requests the key did NOT route through the router and duplicates them through it. reverse answers 'is the router still worth it for a key already on it': it samples the requests the router did serve and duplicates them against baseline_model. The response the caller received is always the real arm

baseline_modelstring nullable

Required when direction is reverse and rejected otherwise: the fixed model the router's own responses are judged against. Must be a plain model rather than another auto-router

shadow_percentagenumber required

Percentage of each target's requests to duplicate through the router

judge_modelstring

Model used to blindly judge real vs. shadow responses. The judge only compares two answers, so a mid-tier model (Claude Sonnet or GPT-4o class) is the sweet spot: small/nano-class models produce unreliable or malformed verdicts, while frontier reasoning models add cost without changing outcomes.

duration_daysinteger

How many days the job samples traffic before completing on its own

max_budgetnumber

Per-target USD budget for the eval's own overhead, the shadow-arm and judge calls, priced with the same figures the spend pipeline bills. EACH scoped target samples until its recorded eval spend reaches this, so a job over N targets spends at most about N times max_budget; in-flight samples can overshoot the cap by one sampling cache window. Every router arm draws from the same per-target budget, so a multi-router job reaches it proportionally sooner

Response

Successful Response

job_idstring required
router_namesstring[] required

Every auto-router this job runs as a shadow arm. Multi-router jobs sample one slice of traffic and judge every arm against the same real responses

modelsstring[]

Model groups the sampled traffic is narrowed to; empty means every model the targets use

direction'forward' | 'reverse'
baseline_modelstring nullable
judge_modelstring required
shadow_percentagenumber required
created_atstring date-time required
ends_atstring date-time required
stopped_bystring nullable

The operator who stopped the job early, recorded by the stop endpoint; 'unknown' backfilled by migration for jobs that displayed stopped when the column arrived; None when the job ended on its own. Its presence is what makes a job read stopped rather than completed

judged_countinteger nullable

Verdicts recorded; detail endpoint only

error_countinteger nullable

Sampled attempts that errored; detail endpoint only

judge_spendnumber nullable

Judge cost so far; detail endpoint only

last_errorstring nullable

Most recent attempt error; detail endpoint only

router_namestring required

The first router, kept for callers that predate router_names; derived so the two fields can never disagree.

status'running' | 'completed' | 'stopped' required

Three recorded facts, no history-guessing: a stop is stopped_by (the migration backfills it for every job that displayed stopped when the column arrived, so the pre-column population is closed), completion is the window passing or every target spending its budget, and anything else is running. The all-targets-stamped fallback covers only stops written by pre-column pods during a rolling deploy.

Changes