--- title: "List benchmark runs" method: GET path: "/factory/{factory_uid}/benchmarks/{suite_uid}/runs" tags: ["Factories", "Benchmarks"] --- # List benchmark runs `GET /factory/{factory_uid}/benchmarks/{suite_uid}/runs` ## Request - Base URL: `https://app.warp.dev/api/v1` - URL: `https://app.warp.dev/api/v1/factory/{factory_uid}/benchmarks/{suite_uid}/runs` - Auth: HTTP bearer List runs launched from this benchmark suite. Each entry contains the run's status and execution/scoring progress. Use `include` to add launch settings, aggregate results, or the creator's display name to each entry. The response contains a page of runs in `data` and continuation information in `page_info`. Retained runs remain readable after the suite is deleted; an unknown suite returns an empty list. ## Path parameters - `factory_uid` string, required - `suite_uid` string, required ## Query parameters - `include` string[] - `state` BenchmarkRunState[] - `limit` integer - `cursor` string ## Response `200` Runs from the frozen suite identity, including deleted suites; empty for an unknown suite - BenchmarkRunList - `data` BenchmarkRun[], required: complete requested Run rows on this page, newest first. - `display_name` string: Optional display name. Surrounding whitespace is trimmed; empty clears the name. - `uid` string, required: existing benchmark Run UID, including legacy non-UUID identities. - `factory_uid` string, required: authorized owning Factory UID. - `factory_version_sha` string: The commit SHA identifying the factory version at the time the benchmark run was launched. Note: This currently does not ensure that the factory configuration stays fixed for the duration of the benchmark run. - `suite_uid` string, required: suite UID verified against the frozen Run context. - `suite_name` string, nullable, required: retained frozen suite name; unknown legacy name is null. - `agent_uid` string, nullable, required: tested root agent's frozen UID when retained. - `agent_name` string, nullable, required: tested root agent's frozen name, not today's catalog name. - `state` 'pending' | 'running' | 'scoring' | 'completed' | 'failed' | 'cancelled', required: Lifecycle state of a benchmark run. - `created_at` string, date-time, required: stored Run creation time. - `updated_at` string, date-time, required: stored Run update time, not a complete source-validity watermark. - `started_at` string, date-time, nullable, required: stored Run start time; null before start or if not retained. - `completed_at` string, date-time, nullable, required: stored Run terminal time; null while active or if not retained. - `creator` BenchmarkRunCreator, required - `type` 'user' | 'agent', required: resolved creator type; agent includes identities behind API keys. - `uid` string, required: creator's public UID, never its database ID. - `display_name` string, nullable: present only for an authorized creator-display-name include; null if the name is unknown, with no email fallback. - `execution` BenchmarkExecutionProgress, required - `planned_total` integer, required: frozen selected task/configuration/repetition matrix size for this scope. - `total` integer, required: observed stored logical Trial row count. - `pending` integer, required: Trials without a currently executing attempt, including queued/claimed attempts. - `running` integer, required: Trials with a currently executing attempt. - `succeeded` integer, required: successfully completed logical Trials. - `failed` integer, required: failed logical Trials after existing retry rules. - `cancelled` integer, required: cancelled logical Trials. - `completed` integer, required: `succeeded + failed + cancelled`, not merely successful Trials. - `scoring` BenchmarkRunScoring, required - `total` integer, nullable, required: all applicable ledger pairs; null before materialization, zero only for known zero work. - `awaiting_trial` integer, required: raw pairs still waiting for execution to finish. - `pending` integer, required: raw pairs awaiting judge dispatch after becoming eligible. - `judging` integer, required: raw pairs with an active judge. - `scored` integer, required: raw pairs marked scored in the durable ledger, even if bound evidence is now absent. - `finished_unscored` integer, required: raw pairs settled without a score. - `not_scoreable` integer, required: raw pairs skipped because execution failed or was cancelled. - `received` integer, required: exactly raw `scored`, not effective retained-score coverage. - `expected` integer, nullable, required: `pending + judging + scored + finished_unscored` once materialized; otherwise null. - `settled` integer, required: raw `scored + finished_unscored + not_scoreable`. - `trials_being_judged` integer, required: succeeded Trials with at least one pending/judging applicable pair. - `trials_scoring_finished` integer, required: succeeded Trials with applicable work all scored/finished-unscored. - `details` BenchmarkRunDetails - `configurations` BenchmarkRecordedConfiguration[], required: complete frozen configurations in launch order, including tooltip settings. - `uid` string, required: stable Run-local public configuration identity, without exposing its stored integer ID. - `display_name` string, required: frozen user-supplied name; the existing empty string means no supplied name. - `harness` string, nullable, required: recorded effective root harness; unknown legacy values remain null. - `model` string, nullable, required: recorded effective root model; never filled from live defaults. - `runner_id` string, nullable, required: recorded root runner identity; no per-agent runner is introduced. - `harness_auth_secret_name` string, nullable, required: retained root credential-name reference, including unavailable names. - `agent_configs` BenchmarkRecordedAgentConfiguration[], required: complete retained named-agent settings, including the root when recorded. - `agent_uid` string, required: frozen named Factory agent UID. - `agent_name` string, nullable, required: retained frozen named-agent label. - `harness` string, nullable, required: named agent's recorded effective harness. - `model` string, nullable, required: named agent's recorded effective model. - `harness_auth_secret_name` string, nullable, required: retained named-agent credential-name reference. - `task_uids` string[], required: complete selected frozen task identities in launch order, including derived legacy identities. - `repetition_count` integer, required: frozen repetitions per task/configuration. - `scorer_selection` BenchmarkScorerSelection, required - `mode` 'all_applicable' | 'selected', required: preserved launch selection mode, even if selected scorers later disappear. - `scorer_uids` string[], required: complete known canonical scorer UUIDs in the selected allowlist; empty for all-applicable. - `scorers` BenchmarkScorer[], required: safe descriptors for this Run's resolved scorers, using the existing authorized resolver. - `scorer_uid` string, required: canonical scorer UUID supplied by the integrated identity adapter. - `name` string, required: existing resolved scorer name, not a promised launch-frozen name. - `kind` 'user' | 'benchmark_task_correctness', required: Whether the scorer is user-defined or a platform-owned managed scorer. Read-only on responses; create input does not accept scorer_kind. - `output_type` 'classification', required: current benchmark results support classification only. - `output_labels` string[], required: existing resolved classification label values in configuration order. - `tags` BenchmarkTagTasks[], required: complete unfiltered frozen tag-to-task mapping; derive each tag's task count from its UID-array length. - `tag` string, required: exact frozen tag value, ordered lexically. - `task_uids` string[], required: complete distinct selected frozen task UIDs carrying this tag, in the containing `details.task_uids` order. - `summary_results` BenchmarkResultsSummary - `factory_uid` string, required: authorized owning Factory UID. - `suite_uid` string, required: verified frozen suite UID. - `run_uid` string, required: containing benchmark Run UID. - `state` 'pending' | 'running' | 'scoring' | 'completed' | 'failed' | 'cancelled', required: Lifecycle state of a benchmark run. - `as_of` string, date-time, required: calculation time for the agreed on-request summary; not a source snapshot or provider-final time. - `provisional` boolean, required: true for pending/running/scoring Runs whose values can still change. - `incomplete_reason` string, nullable, required: existing safe failed/cancelled partial-results explanation; null in other states. - `coverage` BenchmarkResultsCoverage, required - `execution` BenchmarkExecutionCoverage, required - `planned_trial_count` integer, required: selected frozen matrix size, equal to `execution.planned_total`. - `materialized_trial_count` integer, required: observed selected logical Trial rows, equal to `execution.total`. - `scoring` BenchmarkScoringCoverage, required - `effective_scored_pair_count` integer, required: pairs whose exact bound score resolves under retention/access and at/before cutoff. - `expected_pair_count` integer, nullable, required: materialized `pending + judging + scored + finished_unscored`; excludes awaiting-trial/not-scoreable and is null before membership is known. - `compute_credits` string, required: existing selected execution compute-credit total from RunMetrics. - `platform_credits` string, required: existing selected execution platform-credit total from RunMetrics. - `spend` BenchmarkSpend, required - `execution` BenchmarkSpendTotal, required - `cost_usd` number, double, required: existing mixed billed/fallback/provider-estimate dollar total. - `total_credits` string, required: existing Warp credit total for this spend group. - `judging` BenchmarkSpendTotal, required - `cost_usd` number, double, required: existing mixed billed/fallback/provider-estimate dollar total. - `total_credits` string, required: existing Warp credit total for this spend group. - `combined` BenchmarkSpendTotal, required - `cost_usd` number, double, required: existing mixed billed/fallback/provider-estimate dollar total. - `total_credits` string, required: existing Warp credit total for this spend group. - `elapsed_s` number, double, nullable, required: Run span from start through completion/calculation time, including scoring; null before start or if start is unknown. - `task_count` integer, required: distinct selected frozen tasks, independent of display projections and Trial pages. - `scorer_count` integer, nullable, required: count of resolved scorer descriptors, as in current history statistics; null before any selected Trial dispatch. - `narrative` BenchmarkNarrative, required - `availability` 'pending' | 'available' | 'unavailable', required: current stored generation/lifecycle gate, not metric completeness. - `text` string, nullable, required: nonempty retained narrative only when available; null otherwise. - `page_info` BenchmarkPageInfo, required - `has_next_page` boolean, required: true when another complete list page exists. - `next_cursor` string: opaque scoped keyset cursor, present only when `has_next_page=true`, never null. ## Other responses - `401`: Authentication required - `403`: Factory access is not enabled for the authenticated principal - `404`: Factory not found or not accessible ## Changes - **2026-11-04** `4b5d145669be`: 1 info - added the optional property `data/items/display_name` to the response with the `200` status - **2026-10-21** `7ee3ba55debc`: 3 info - api tag `factory-benchmarks` added - api tag `benchmark` removed - api tag `factory` removed - **2026-10-19** `2c45dcc0a13e`: 2 breaking, 2 info - added `#/components/schemas/BenchmarkRunCreator, subschema #2` to the `data/items/creator` response property `oneOf` list for the response status `200` - added `#/components/schemas/BenchmarkSpend, subschema #2` to the `data/items/summary_results/spend` response property `oneOf` list for the response status `200` - removed `#/components/schemas/BenchmarkRunCreator` from the `data/items/creator` response property `allOf` list for the response status `200` - removed `#/components/schemas/BenchmarkSpend` from the `data/items/summary_results/spend` response property `allOf` list for the response status `200` - **2026-10-15** `f4630aae17ba`: 1 info - endpoint added [Operation history](https://skmtc.dev/warp/apis/oz-agent-api/changes/factory/:factory_uid/benchmarks/:suite_uid/runs/get.md) --- [API](https://skmtc.dev/warp/apis/oz-agent-api.md) - [All operations](https://skmtc.dev/warp/apis/oz-agent-api/llms.txt) - [OpenAPI document](https://skmtc.dev/warp/apis/oz-agent-api/revisions/4b5d145669be?raw)