---
title: "Start Shadow Eval"
method: POST
path: "/auto_router/shadow_eval/start"
tags: ["auto router"]
---

# Start Shadow Eval

`POST /auto_router/shadow_eval/start`

Start a shadow eval: duplicate a sampled slice of one or more targets' live traffic
against a second arm, judge the two responses blind, and stratify win rates by tier,
by the model that served the real arm, and by target.

A target is a virtual key, a team, or a user. Team and user targets match on the
identity every request resolves to at auth time, so they cover JWT-authenticated
traffic, which presents no virtual key; a user target samples that user's traffic
across all their teams, whether it arrives on a JWT or a key they own. models narrows
every target to requests for those model groups, so a user plus one model samples that
user's traffic on that model across every key they own; it is forward-only, since a
reverse job already samples exactly the traffic its own router served.

A forward job answers whether the targets should adopt router_name: it samples the
requests the router did not serve and duplicates them through it. A reverse job
answers whether a target already on the router still gains from it: it samples the
requests the router did serve and duplicates them against baseline_model. A target
can hold one active job per direction, so both questions can run at once, and a
request matching several jobs' targets (say its key and its team) is sampled by
each, separately budgeted.

Shadow responses are never served to users. Each target samples until its recorded
eval spend, the shadow and judge calls' own cost, reaches max_budget dollars, the
job's window ends, or the job is stopped, so one target running out of budget does
not end sampling for the others; sampling changes propagate to pods within about 10
seconds. Shadow and judge calls bill to the sampled request's own identity but are
excluded from request counts and auto-router adoption metrics.

## Request body

- StartShadowEvalRequest — Start duplicating one or more targets' traffic for blind comparison against an auto-router. A target is a virtual key, a team, or a user; each becomes its own leg with its own budget and stop state. Team and user targets match on the identity every request carries after auth (user_api_key_team_id / user_api_key_user_id), so they cover JWT-authenticated traffic, which presents no virtual key at all.
  - `api_key_ids` string[] — Hashed virtual keys whose traffic will be shadowed. Combined with team_ids and user_ids the job needs at least one target and at most 100, which also bounds every read the job's endpoints make. Each target carries its own max_budget spend budget, so one exhausting its budget leaves the others sampling.
  - `team_ids` string[] — Teams whose traffic will be shadowed, matched on the team every authenticated request resolves to, so a team's JWT-auth and virtual-key traffic are both sampled
  - `user_ids` string[] — Users whose traffic will be shadowed, matched on the user every authenticated request resolves to across all their teams: JWT requests carrying their subject claim and virtual keys they own
  - `models` string[] — Model groups to narrow the sampled traffic to, matched on the group the caller requested and resolved through model_group_alias, so an alias and its target are one name. Empty samples every model the targets use. This ANDs with the targets: a job over a user and one model samples that user's requests on that model across every key they own, and none of their other traffic. Forward jobs only: a reverse job samples exactly the traffic its own router served, which no other model group can name
  - `router_name` string, nullable — The auto-router under evaluation, in either direction: the single-router spelling of router_names. Provide exactly one of the two fields
  - `router_names` string[] — The auto-routers under evaluation, at most 4. Every sampled request runs through every router listed and each arm is judged independently against the same real response, so routers compare head-to-head on identical traffic. More than one router requires direction 'forward'. After validation this field always carries the full deduplicated set, whichever spelling the caller used
  - `direction` 'forward' | 'reverse' — forward answers 'should this key adopt router_name': it samples the requests the key did NOT route through the router and duplicates them through it. reverse answers 'is the router still worth it for a key already on it': it samples the requests the router did serve and duplicates them against baseline_model. The response the caller received is always the real arm
  - `baseline_model` string, nullable — Required when direction is reverse and rejected otherwise: the fixed model the router's own responses are judged against. Must be a plain model rather than another auto-router
  - `shadow_percentage` number, required — Percentage of each target's requests to duplicate through the router
  - `judge_model` string — Model used to blindly judge real vs. shadow responses. The judge only compares two answers, so a mid-tier model (Claude Sonnet or GPT-4o class) is the sweet spot: small/nano-class models produce unreliable or malformed verdicts, while frontier reasoning models add cost without changing outcomes.
  - `duration_days` integer — How many days the job samples traffic before completing on its own
  - `max_budget` number — Per-target USD budget for the eval's own overhead, the shadow-arm and judge calls, priced with the same figures the spend pipeline bills. EACH scoped target samples until its recorded eval spend reaches this, so a job over N targets spends at most about N times max_budget; in-flight samples can overshoot the cap by one sampling cache window. Every router arm draws from the same per-target budget, so a multi-router job reaches it proportionally sooner

## Response `201`

Successful Response

- ShadowEvalJobResponse — A shadow-eval job over one or more targets, each with its own budget and stop state; status is derived from stopped_by, the targets' stop and budget state, and ends_at, never stored, so no writer anywhere can produce an inconsistent one. Aggregate fields are populated by the detail endpoint only and stay None on list responses.
  - `job_id` string, required
  - `targets` ShadowEvalJobTargetResponse[], required — The targets whose traffic this job evaluates, and only theirs, each with its own budget
    - `target_type` 'key' | 'team' | 'user', required — What kind of entity this entry scopes
    - `target_id` string, required — The hashed virtual key, team id, or user id whose traffic this entry scopes
    - `max_turns` integer, required — This target's sample-count ceiling: the whole budget for jobs created before max_budget existed, and the error-loop safety valve otherwise
    - `max_budget` number, nullable — This target's own USD budget for the eval's shadow and judge spend, independent of its siblings'; None on jobs created before spend budgets existed, which max_turns alone bounds
    - `stopped_at` string, date-time, nullable — When this target's slot was stamped free, whether its own budget ran out, the window closed, or an operator stopped the job; status is derived, so a spent budget reads completed even while this is still unset
    - `attempt_count` integer, nullable — This target's sampled attempts so far, judged and errored alike, the same count the sampler budgets against max_turns; populated on list and detail responses. Frozen at stopped_at once the target is stamped, so in-flight attempts landing after a stop never reclassify it
    - `spend` number, nullable — This target's recorded shadow plus judge spend in USD, the same figure the sampler budgets against max_budget; populated on list and detail responses and frozen at stopped_at exactly like attempt_count
    - `verdicts` ShadowEvalSlice — Judge outcomes for one slice of a job's verdicts: a router tier, one of the models that served the real arm, or one scoped target (embedded on that target's own entry, so slices never need re-joining to a target by id).
      - `group` string, required
      - `turn_count` integer, required
      - `real_win_rate_pct` number, required — Share of judged turns the real arm won, meaning the response the caller actually received: the key's own model in forward mode, the router's pick in reverse
      - `shadow_win_rate_pct` number, required — Share of judged turns the shadow arm won, meaning the duplicated response nobody was served: the router's pick in forward mode, baseline_model in reverse
      - `tie_rate_pct` number, required
      - `avg_judge_confidence` number, required
      - `real_spend` number — USD the real arm billed on this slice's judged turns, completion plus its own routing classifier when it routed, excluding turns litellm's response cache served for free
      - `shadow_spend` number — USD the shadow arm billed on the same turns, completion plus its own routing classifier, excluding the judge and the same cache-served turns, so the two spends compare like for like
      - `cache_hit_turns` integer — Judged turns litellm's response cache served, excluded from both spends: an adopted router would be served by the same cache, so those turns cost the same either way
    - `target_alias` string, nullable — Display label resolved from the target's own row at read time: the key's alias, the team's alias, or the user's email; None when unset or deleted
    - `key_name` string, nullable — Masked display name (sk-...) for key targets, resolved at read time; None for teams and users
  - `router_names` string[], required — Every auto-router this job runs as a shadow arm. Multi-router jobs sample one slice of traffic and judge every arm against the same real responses
  - `models` string[] — Model groups the sampled traffic is narrowed to; empty means every model the targets use
  - `direction` 'forward' | 'reverse'
  - `baseline_model` string, nullable
  - `judge_model` string, required
  - `shadow_percentage` number, required
  - `created_at` string, date-time, required
  - `ends_at` string, date-time, required
  - `stopped_by` string, nullable — The operator who stopped the job early, recorded by the stop endpoint; 'unknown' backfilled by migration for jobs that displayed stopped when the column arrived; None when the job ended on its own. Its presence is what makes a job read stopped rather than completed
  - `judged_count` integer, nullable — Verdicts recorded; detail endpoint only
  - `error_count` integer, nullable — Sampled attempts that errored; detail endpoint only
  - `judge_spend` number, nullable — Judge cost so far; detail endpoint only
  - `last_error` string, nullable — Most recent attempt error; detail endpoint only
  - `results` ShadowEvalResult — Stratified results of a shadow-eval job's verdicts so far.
    - `by_tier` ShadowEvalSlice[], required
      - `group` string, required
      - `turn_count` integer, required
      - `real_win_rate_pct` number, required — Share of judged turns the real arm won, meaning the response the caller actually received: the key's own model in forward mode, the router's pick in reverse
      - `shadow_win_rate_pct` number, required — Share of judged turns the shadow arm won, meaning the duplicated response nobody was served: the router's pick in forward mode, baseline_model in reverse
      - `tie_rate_pct` number, required
      - `avg_judge_confidence` number, required
      - `real_spend` number — USD the real arm billed on this slice's judged turns, completion plus its own routing classifier when it routed, excluding turns litellm's response cache served for free
      - `shadow_spend` number — USD the shadow arm billed on the same turns, completion plus its own routing classifier, excluding the judge and the same cache-served turns, so the two spends compare like for like
      - `cache_hit_turns` integer — Judged turns litellm's response cache served, excluded from both spends: an adopted router would be served by the same cache, so those turns cost the same either way
    - `by_current_model` ShadowEvalSlice[], required — Sliced by the model that served the real arm: the keys' incumbent models in forward mode, and in reverse the models the router itself picked
      - `group` string, required
      - `turn_count` integer, required
      - `real_win_rate_pct` number, required — Share of judged turns the real arm won, meaning the response the caller actually received: the key's own model in forward mode, the router's pick in reverse
      - `shadow_win_rate_pct` number, required — Share of judged turns the shadow arm won, meaning the duplicated response nobody was served: the router's pick in forward mode, baseline_model in reverse
      - `tie_rate_pct` number, required
      - `avg_judge_confidence` number, required
      - `real_spend` number — USD the real arm billed on this slice's judged turns, completion plus its own routing classifier when it routed, excluding turns litellm's response cache served for free
      - `shadow_spend` number — USD the shadow arm billed on the same turns, completion plus its own routing classifier, excluding the judge and the same cache-served turns, so the two spends compare like for like
      - `cache_hit_turns` integer — Judged turns litellm's response cache served, excluded from both spends: an adopted router would be served by the same cache, so those turns cost the same either way
    - `by_router` ShadowEvalSlice[] — One slice per router arm, grouped on the router name. Every arm of a multi-router job is judged against the same real responses over the same sampled requests, so these slices compare routers head-to-head: like-for-like win rates and spends on identical traffic. Verdicts from before arm stamping existed count toward the job's own router
      - `group` string, required
      - `turn_count` integer, required
      - `real_win_rate_pct` number, required — Share of judged turns the real arm won, meaning the response the caller actually received: the key's own model in forward mode, the router's pick in reverse
      - `shadow_win_rate_pct` number, required — Share of judged turns the shadow arm won, meaning the duplicated response nobody was served: the router's pick in forward mode, baseline_model in reverse
      - `tie_rate_pct` number, required
      - `avg_judge_confidence` number, required
      - `real_spend` number — USD the real arm billed on this slice's judged turns, completion plus its own routing classifier when it routed, excluding turns litellm's response cache served for free
      - `shadow_spend` number — USD the shadow arm billed on the same turns, completion plus its own routing classifier, excluding the judge and the same cache-served turns, so the two spends compare like for like
      - `cache_hit_turns` integer — Judged turns litellm's response cache served, excluded from both spends: an adopted router would be served by the same cache, so those turns cost the same either way
    - `overall_shadow_win_rate_pct` number, required
    - `overall_tie_rate_pct` number, required
    - `sampled_real_spend` number — USD the real arm billed across all judged turns, cache-served turns excluded. A judged turn is one (request, router arm) verdict, so a multi-router job counts the real response once per arm it was judged against; per-router comparisons read by_router
    - `sampled_shadow_spend` number — USD the shadow arms billed across the same turns, judge excluded, like for like
    - `not_sampled_count` integer, nullable — Eligible requests the sampling dice skipped, summed over legs: the judged rows stand for judged + this many requests. None for jobs from before the funnel existed
    - `unjudgeable_count` integer, nullable — Sampled requests whose shape could not be judged (tool-final turn, empty text)
    - `shed_count` integer, nullable — Sampled requests dropped by the per-pod concurrency cap, so quiet periods are overweighted
    - `withheld_count` integer, nullable — Sampled requests the pipeline declined to spend on: no database to record into, an over-budget key or team, or the eval budget unverifiable or already reached (the in-flight burst as a job crosses max_budget lands here rather than vanishing from coverage)
  - `router_name` string, required — The first router, kept for callers that predate router_names; derived so the two fields can never disagree.
  - `status` 'running' | 'completed' | 'stopped', required — Three recorded facts, no history-guessing: a stop is stopped_by (the migration backfills it for every job that displayed stopped when the column arrived, so the pre-column population is closed), completion is the window passing or every target spending its budget, and anything else is running. The all-targets-stamped fallback covers only stops written by pre-column pods during a rolling deploy.

## Other responses

- `422` — Validation Error

## Changes

- **2026-09-18** `082b5fabd909` — 1 info
  - endpoint added

[Change history](https://skmtc.dev/flock/apis/litellm-api/changes/auto_router/shadow_eval/start/post.md)

---

[API](https://skmtc.dev/flock/apis/litellm-api.md) · [All operations](https://skmtc.dev/flock/apis/litellm-api/llms.txt) · [OpenAPI document](https://skmtc.dev/flock/apis/litellm-api/revisions/c71315002afa?raw)