WAwarp
FactoriesBenchmarks

Launch a benchmark run from the current suite version

Changed on

Each successful request creates a new run. Launch is not idempotent: disable automatic retries for this request and reconcile an uncertain outcome by listing runs before launching again. Retries are a client transport option, not a request-body field. In the TypeScript SDK, pass { maxRetries: 0 } as the request options. In the Python SDK, call create through client.with_options(max_retries=0). These options do not change execution retries within the benchmark run.

post/factory/{factory_uid}/benchmarks/{suite_uid}/runs

Request

  • Base URL
    https://app.warp.dev/api/v1
  • Auth
    HTTP bearer

Path parameters

factory_uidstring required

UID of the factory that owns the benchmark suite.

suite_uidstring required

UID of the benchmark suite within this factory.

Request body

display_namestring

Optional display name. Surrounding whitespace is trimmed; empty clears the name.

repetition_countinteger required

Number of repetitions per (task, configuration) pair, frozen onto the run. Like configurations and scorer_selection, this is chosen fresh at each launch rather than persisted on the suite.

tasksstring[]

Optional allowlist of immutable UIDs for current tasks in the addressed suite. Omitted runs all current suite tasks. Explicit empty, invalid, unknown, stale, cross-suite, and duplicate selections are rejected.

Example request

{
  "tasks": [
    "7d69fb52-71c5-4e27-914e-8764ee9c5246",
    "ad8ff7dc-a7d1-4ed6-9f50-650010bb39aa"
  ]
}

Response

Run launched

uidstring required
state'pending' | 'running' | 'scoring' | 'completed' | 'failed' | 'cancelled' required

Lifecycle state of a benchmark run.

Changes

  • 4b5d145669beRevision changes
    • warning

      the // request property's maxLength was set to 200

    • info

      added the new optional request property

  • 7ee3ba55debcRevision changes
    • info

      api tag factory-benchmarks added

    • info

      api tag benchmark removed

    • info

      api tag factory removed