---
title: "List Shadow Eval Jobs"
method: GET
path: "/auto_router/shadow_eval"
tags: ["auto router"]
---

# List Shadow Eval Jobs

`GET /auto_router/shadow_eval`

List shadow eval jobs, newest first, each target with its attempt count so status
is accurate. Judged counts, spend, and results ride the detail endpoint only.

## Query parameters

- `target_type` 'key' | 'team' | 'user', nullable — Kind of target to filter on; requires target_id
- `target_id` string, nullable — Filter to jobs that shadow this target, alone or alongside others
- `limit` integer — Newest jobs to return

## Response `200`

Successful Response

- ShadowEvalJobResponse[]
  - `job_id` string, required
  - `targets` ShadowEvalJobTargetResponse[], required — The targets whose traffic this job evaluates, and only theirs, each with its own budget
    - `target_type` 'key' | 'team' | 'user', required — What kind of entity this entry scopes
    - `target_id` string, required — The hashed virtual key, team id, or user id whose traffic this entry scopes
    - `max_turns` integer, required — This target's sample-count ceiling: the whole budget for jobs created before max_budget existed, and the error-loop safety valve otherwise
    - `max_budget` number, nullable — This target's own USD budget for the eval's shadow and judge spend, independent of its siblings'; None on jobs created before spend budgets existed, which max_turns alone bounds
    - `stopped_at` string, date-time, nullable — When this target's slot was stamped free, whether its own budget ran out, the window closed, or an operator stopped the job; status is derived, so a spent budget reads completed even while this is still unset
    - `attempt_count` integer, nullable — This target's sampled attempts so far, judged and errored alike, the same count the sampler budgets against max_turns; populated on list and detail responses. Frozen at stopped_at once the target is stamped, so in-flight attempts landing after a stop never reclassify it
    - `spend` number, nullable — This target's recorded shadow plus judge spend in USD, the same figure the sampler budgets against max_budget; populated on list and detail responses and frozen at stopped_at exactly like attempt_count
    - `verdicts` ShadowEvalSlice — Judge outcomes for one slice of a job's verdicts: a router tier, one of the models that served the real arm, or one scoped target (embedded on that target's own entry, so slices never need re-joining to a target by id).
      - `group` string, required
      - `turn_count` integer, required
      - `real_win_rate_pct` number, required — Share of judged turns the real arm won, meaning the response the caller actually received: the key's own model in forward mode, the router's pick in reverse
      - `shadow_win_rate_pct` number, required — Share of judged turns the shadow arm won, meaning the duplicated response nobody was served: the router's pick in forward mode, baseline_model in reverse
      - `tie_rate_pct` number, required
      - `avg_judge_confidence` number, required
      - `real_spend` number — USD the real arm billed on this slice's judged turns, completion plus its own routing classifier when it routed, excluding turns litellm's response cache served for free
      - `shadow_spend` number — USD the shadow arm billed on the same turns, completion plus its own routing classifier, excluding the judge and the same cache-served turns, so the two spends compare like for like
      - `cache_hit_turns` integer — Judged turns litellm's response cache served, excluded from both spends: an adopted router would be served by the same cache, so those turns cost the same either way
    - `target_alias` string, nullable — Display label resolved from the target's own row at read time: the key's alias, the team's alias, or the user's email; None when unset or deleted
    - `key_name` string, nullable — Masked display name (sk-...) for key targets, resolved at read time; None for teams and users
  - `router_names` string[], required — Every auto-router this job runs as a shadow arm. Multi-router jobs sample one slice of traffic and judge every arm against the same real responses
  - `models` string[] — Model groups the sampled traffic is narrowed to; empty means every model the targets use
  - `direction` 'forward' | 'reverse'
  - `baseline_model` string, nullable
  - `judge_model` string, required
  - `shadow_percentage` number, required
  - `created_at` string, date-time, required
  - `ends_at` string, date-time, required
  - `stopped_by` string, nullable — The operator who stopped the job early, recorded by the stop endpoint; 'unknown' backfilled by migration for jobs that displayed stopped when the column arrived; None when the job ended on its own. Its presence is what makes a job read stopped rather than completed
  - `judged_count` integer, nullable — Verdicts recorded; detail endpoint only
  - `error_count` integer, nullable — Sampled attempts that errored; detail endpoint only
  - `judge_spend` number, nullable — Judge cost so far; detail endpoint only
  - `last_error` string, nullable — Most recent attempt error; detail endpoint only
  - `results` ShadowEvalResult — Stratified results of a shadow-eval job's verdicts so far.
    - `by_tier` ShadowEvalSlice[], required
      - `group` string, required
      - `turn_count` integer, required
      - `real_win_rate_pct` number, required — Share of judged turns the real arm won, meaning the response the caller actually received: the key's own model in forward mode, the router's pick in reverse
      - `shadow_win_rate_pct` number, required — Share of judged turns the shadow arm won, meaning the duplicated response nobody was served: the router's pick in forward mode, baseline_model in reverse
      - `tie_rate_pct` number, required
      - `avg_judge_confidence` number, required
      - `real_spend` number — USD the real arm billed on this slice's judged turns, completion plus its own routing classifier when it routed, excluding turns litellm's response cache served for free
      - `shadow_spend` number — USD the shadow arm billed on the same turns, completion plus its own routing classifier, excluding the judge and the same cache-served turns, so the two spends compare like for like
      - `cache_hit_turns` integer — Judged turns litellm's response cache served, excluded from both spends: an adopted router would be served by the same cache, so those turns cost the same either way
    - `by_current_model` ShadowEvalSlice[], required — Sliced by the model that served the real arm: the keys' incumbent models in forward mode, and in reverse the models the router itself picked
      - `group` string, required
      - `turn_count` integer, required
      - `real_win_rate_pct` number, required — Share of judged turns the real arm won, meaning the response the caller actually received: the key's own model in forward mode, the router's pick in reverse
      - `shadow_win_rate_pct` number, required — Share of judged turns the shadow arm won, meaning the duplicated response nobody was served: the router's pick in forward mode, baseline_model in reverse
      - `tie_rate_pct` number, required
      - `avg_judge_confidence` number, required
      - `real_spend` number — USD the real arm billed on this slice's judged turns, completion plus its own routing classifier when it routed, excluding turns litellm's response cache served for free
      - `shadow_spend` number — USD the shadow arm billed on the same turns, completion plus its own routing classifier, excluding the judge and the same cache-served turns, so the two spends compare like for like
      - `cache_hit_turns` integer — Judged turns litellm's response cache served, excluded from both spends: an adopted router would be served by the same cache, so those turns cost the same either way
    - `by_router` ShadowEvalSlice[] — One slice per router arm, grouped on the router name. Every arm of a multi-router job is judged against the same real responses over the same sampled requests, so these slices compare routers head-to-head: like-for-like win rates and spends on identical traffic. Verdicts from before arm stamping existed count toward the job's own router
      - `group` string, required
      - `turn_count` integer, required
      - `real_win_rate_pct` number, required — Share of judged turns the real arm won, meaning the response the caller actually received: the key's own model in forward mode, the router's pick in reverse
      - `shadow_win_rate_pct` number, required — Share of judged turns the shadow arm won, meaning the duplicated response nobody was served: the router's pick in forward mode, baseline_model in reverse
      - `tie_rate_pct` number, required
      - `avg_judge_confidence` number, required
      - `real_spend` number — USD the real arm billed on this slice's judged turns, completion plus its own routing classifier when it routed, excluding turns litellm's response cache served for free
      - `shadow_spend` number — USD the shadow arm billed on the same turns, completion plus its own routing classifier, excluding the judge and the same cache-served turns, so the two spends compare like for like
      - `cache_hit_turns` integer — Judged turns litellm's response cache served, excluded from both spends: an adopted router would be served by the same cache, so those turns cost the same either way
    - `overall_shadow_win_rate_pct` number, required
    - `overall_tie_rate_pct` number, required
    - `sampled_real_spend` number — USD the real arm billed across all judged turns, cache-served turns excluded. A judged turn is one (request, router arm) verdict, so a multi-router job counts the real response once per arm it was judged against; per-router comparisons read by_router
    - `sampled_shadow_spend` number — USD the shadow arms billed across the same turns, judge excluded, like for like
    - `not_sampled_count` integer, nullable — Eligible requests the sampling dice skipped, summed over legs: the judged rows stand for judged + this many requests. None for jobs from before the funnel existed
    - `unjudgeable_count` integer, nullable — Sampled requests whose shape could not be judged (tool-final turn, empty text)
    - `shed_count` integer, nullable — Sampled requests dropped by the per-pod concurrency cap, so quiet periods are overweighted
    - `withheld_count` integer, nullable — Sampled requests the pipeline declined to spend on: no database to record into, an over-budget key or team, or the eval budget unverifiable or already reached (the in-flight burst as a job crosses max_budget lands here rather than vanishing from coverage)
  - `router_name` string, required — The first router, kept for callers that predate router_names; derived so the two fields can never disagree.
  - `status` 'running' | 'completed' | 'stopped', required — Three recorded facts, no history-guessing: a stop is stopped_by (the migration backfills it for every job that displayed stopped when the column arrived, so the pre-column population is closed), completion is the window passing or every target spending its budget, and anything else is running. The all-targets-stamped fallback covers only stops written by pre-column pods during a rolling deploy.

## Other responses

- `422` — Validation Error

## Changes

- **2026-09-18** `082b5fabd909` — 1 info
  - endpoint added

[Change history](https://skmtc.dev/flock/apis/litellm-api/changes/auto_router/shadow_eval/get.md)

---

[API](https://skmtc.dev/flock/apis/litellm-api.md) · [All operations](https://skmtc.dev/flock/apis/litellm-api/llms.txt) · [OpenAPI document](https://skmtc.dev/flock/apis/litellm-api/revisions/c71315002afa?raw)