--- title: "[Beta] Create a BenchmarkJob." method: POST path: "/v1/benchmark_jobs" tags: ["Benchmark"] --- # [Beta] Create a BenchmarkJob. `POST /v1/benchmark_jobs` [Beta] Create a BenchmarkJob that runs a set of scenarios entirely on runloop. ## Request body - BenchmarkJobCreateParameters — BenchmarkJobCreateParameters contain the set of parameters to create a BenchmarkJob. - `name` string, nullable — The name of the BenchmarkJob. If not provided, name will be generated based on target dataset. - `spec` union - HarborJobSpec — Harbor-based job specification with inline YAML configuration. - `inline_yaml` string, required — The Harbor job configuration as inline YAML content. - `type` 'harbor', required - BenchmarkDefinitionJobSpec — Specifies a benchmark definition with runtime configuration. The benchmark definition's scenarios will be executed using the provided agent and orchestrator configurations. - `benchmark_id` string, required — ID of the benchmark definition to run. The scenarios from this benchmark will be executed. - `agent_configs` JobAgentConfig[], required — Agent configurations to use for this run. Must specify at least one agent. - `agent_id` string, nullable — ID of the agent to use (optional if agent exists by name) - `name` string, required — Name of the agent - `model_name` string, nullable — Model name override for this agent - `timeout_seconds` number, float, nullable — Timeout in seconds for this agent - `kwargs` object, nullable — Additional kwargs for agent configuration - `agent_environment` JobAgentEnvironment — Environment configuration for an agent in a benchmark job - `environment_variables` object, nullable — Environment variables to set when launching the agent. - `secrets` object, nullable — Secrets to inject as environment variables when launching the agent. Map of environment variable names to secret IDs. - `type` 'job_agent', required - `orchestrator_config` JobOrchestratorConfig — Orchestrator configuration for benchmark job execution - `n_concurrent_trials` integer, nullable — Number of concurrent trials to run (default: 1). Controls parallelism for scenario execution. Default is 1. - `n_attempts` integer, nullable — Number of retry attempts on failure (default: 0). This is the retry policy for failed scenarios. Default is 0. - `timeout_multiplier` number, float, nullable — Timeout multiplier for retries (default: 1.0). Each retry will multiply the timeout by this factor. - `quiet` boolean, nullable — Suppress verbose output (default: false) - `type` 'benchmark', required - ScenarioDefinitionJobSpec — Specifies a set of scenarios with runtime configuration. The scenarios will be executed using the provided agent and orchestrator configurations. - `scenario_ids` string[], required — List of scenario IDs to execute - `agent_configs` JobAgentConfig[], required — Agent configurations to use for this run. Must specify at least one agent. - `agent_id` string, nullable — ID of the agent to use (optional if agent exists by name) - `name` string, required — Name of the agent - `model_name` string, nullable — Model name override for this agent - `timeout_seconds` number, float, nullable — Timeout in seconds for this agent - `kwargs` object, nullable — Additional kwargs for agent configuration - `agent_environment` JobAgentEnvironment — Environment configuration for an agent in a benchmark job - `environment_variables` object, nullable — Environment variables to set when launching the agent. - `secrets` object, nullable — Secrets to inject as environment variables when launching the agent. Map of environment variable names to secret IDs. - `type` 'job_agent', required - `orchestrator_config` JobOrchestratorConfig — Orchestrator configuration for benchmark job execution - `n_concurrent_trials` integer, nullable — Number of concurrent trials to run (default: 1). Controls parallelism for scenario execution. Default is 1. - `n_attempts` integer, nullable — Number of retry attempts on failure (default: 0). This is the retry policy for failed scenarios. Default is 0. - `timeout_multiplier` number, float, nullable — Timeout multiplier for retries (default: 1.0). Each retry will multiply the timeout by this factor. - `quiet` boolean, nullable — Suppress verbose output (default: false) - `type` 'scenarios', required ## Response `200` OK - BenchmarkJobView — A BenchmarkJobView represents a benchmark job that runs a set of scenarios entirely on runloop. - `id` string, required — The ID of the BenchmarkJob. - `name` string, required — The unique name of the BenchmarkJob. - `state` 'initializing' | 'queued' | 'running' | 'completed' | 'failed' | 'cancelled' | 'timeout', required - `job_source` union - HarborJobSource — Harbor job source with inline YAML configuration - `inline_yaml` string, required — The Harbor job configuration as inline YAML content - `type` 'harbor', required - BenchmarkDefJobSource — Benchmark definition job source - `benchmark_id` string, required — The ID of the benchmark definition - `benchmark_name` string, nullable — Optional user-provided name for the benchmark definition - `type` 'benchmark', required - ScenariosJobSource — Scenarios job source with a list of scenario definition IDs - `scenario_ids` string[], required — List of scenario definition IDs to execute - `type` 'scenarios', required - `job_spec` JobSpec — Job specification describing scenarios and execution configuration - `scenario_ids` string[], required — List of scenario IDs to execute - `orchestrator_config` JobOrchestratorConfig — Orchestrator configuration for benchmark job execution - `n_concurrent_trials` integer, nullable — Number of concurrent trials to run (default: 1). Controls parallelism for scenario execution. Default is 1. - `n_attempts` integer, nullable — Number of retry attempts on failure (default: 0). This is the retry policy for failed scenarios. Default is 0. - `timeout_multiplier` number, float, nullable — Timeout multiplier for retries (default: 1.0). Each retry will multiply the timeout by this factor. - `quiet` boolean, nullable — Suppress verbose output (default: false) - `agent_configs` JobAgentConfig[], required — Agent configurations for this job - `agent_id` string, nullable — ID of the agent to use (optional if agent exists by name) - `name` string, required — Name of the agent - `model_name` string, nullable — Model name override for this agent - `timeout_seconds` number, float, nullable — Timeout in seconds for this agent - `kwargs` object, nullable — Additional kwargs for agent configuration - `agent_environment` JobAgentEnvironment — Environment configuration for an agent in a benchmark job - `environment_variables` object, nullable — Environment variables to set when launching the agent. - `secrets` object, nullable — Secrets to inject as environment variables when launching the agent. Map of environment variable names to secret IDs. - `type` 'job_agent', required - `failure_reason` string, nullable — Failure reason if job failed. - `benchmark_outcomes` BenchmarkOutcomeView[], nullable — Detailed outcome data for each benchmark run created by this job. Includes per-agent results and scenario-level details. - `benchmark_run_id` string, required — The ID of the benchmark run. - `agent_name` string, required — The name of the agent configuration used. - `model_name` string, nullable — The model name used by the agent. - `n_completed` integer, required — Number of scenarios that completed successfully. - `n_failed` integer, required — Number of scenarios that failed. - `n_timeout` integer, required — Number of scenarios that timed out. - `average_score` number, float, nullable — Average score across all completed scenarios (0.0 to 1.0). - `duration_ms` integer, nullable — Total duration of the benchmark run in milliseconds. - `scenario_outcomes` ScenarioOutcomeView[], required — Detailed outcomes for each scenario in this benchmark run. - `scenario_run_id` string, nullable — The ID of the scenario run. May be absent if the scenario failed during setup before a run was created. - `scenario_definition_id` string, required — The ID of the scenario definition that was executed. - `scenario_name` string, required — The name of the scenario. - `state` 'COMPLETED' | 'FAILED' | 'TIMEOUT' | 'CANCELED', required - `score` number, float, nullable — The score achieved for this scenario (0.0 to 1.0). Only present if state is COMPLETED. - `duration_ms` integer, nullable — Duration of the scenario execution in milliseconds. - `failure_reason` FailureReason — Information about why a scenario execution failed - `exception_type` string, required — The exception class name (e.g., 'TimeoutException', 'AgentTimeoutError') - `exception_message` string, required — The exception message providing context - `in_progress_runs` InProgressRunView[], nullable — Benchmark runs currently in progress for this job. Shows runs that have not yet completed. - `benchmark_run_id` string, required — The ID of the benchmark run. - `agent_config` union - ExternalApiAgentConfig — Configuration for externally-driven benchmark runs via API - `info` string, nullable — Placeholder for future external agent metadata - `type` 'external_api', required - JobAgentConfig — Configuration for an agent in a benchmark job - `agent_id` string, nullable — ID of the agent to use (optional if agent exists by name) - `name` string, required — Name of the agent - `model_name` string, nullable — Model name override for this agent - `timeout_seconds` number, float, nullable — Timeout in seconds for this agent - `kwargs` object, nullable — Additional kwargs for agent configuration - `agent_environment` JobAgentEnvironment — Environment configuration for an agent in a benchmark job - `environment_variables` object, nullable — Environment variables to set when launching the agent. - `secrets` object, nullable — Secrets to inject as environment variables when launching the agent. Map of environment variable names to secret IDs. - `type` 'job_agent', required - `state` 'running' | 'canceled' | 'completed' | 'failed', required - `start_time_ms` integer, required — Start time (Unix milliseconds). - `duration_ms` integer, nullable — Duration so far in milliseconds. - `create_time_ms` integer, required — Timestamp when job was created (Unix milliseconds). --- [API](https://skmtc.dev/runloopai/apis/runloop-api.md) · [All operations](https://skmtc.dev/runloopai/apis/runloop-api/llms.txt) · [OpenAPI document](https://skmtc-service-production.skmtc.workers.dev/v1/apis/runloopai/runloop-api/revisions/e9dc776053e4/schema)