Benchmark

Start a new BenchmarkRun.

Changed on

Start a new BenchmarkRun based on the provided Benchmark.

post/v1/benchmarks/start_run

Request

  • Base URL: https://api.runloop.ai
  • URL: https://api.runloop.ai/v1/benchmarks/start_run
  • Auth: HTTP bearer

Request body

benchmark_idstring required

ID of the Benchmark to run.

run_namestring nullable

Display name of the run.

metadataobject nullable

User defined metadata to attach to the benchmark run for organization.

Response

OK

idstring required

The ID of the BenchmarkRun.

benchmark_idstring nullable

The ID of the Benchmark definition. Present if run was created from a benchmark definition.

namestring nullable

The name of the BenchmarkRun.

start_time_msinteger required

The time the benchmark run execution started (Unix timestamp milliseconds).

duration_msinteger nullable

The duration for the BenchmarkRun to complete.

state'running' | 'canceled' | 'completed' | 'failed' required
scorenumber float nullable

The final score across the BenchmarkRun, present once completed. Calculated as sum of scenario scores / number of scenario runs.

metadataobject required

User defined metadata to attach to the benchmark run for organization.

purposestring nullable

Purpose of the run.

environment_variablesobject nullable

Environment variables used to run the benchmark.

secrets_providedobject nullable

User secrets used to run the benchmark. Example: {"DB_PASS": "DATABASE_PASSWORD"} would set the environment variable 'DB_PASS' on all scenario devboxes to the value of the secret 'DATABASE_PASSWORD'.

Changes

  • e9dc776053e4Revision changes
    • info

      added the new pi_json enum value to the request property ///////////////

  • fe489479c0aaRevision changes
    • info

      added the new codex_json enum value to the request property ///////////////