---
title: "Get scenario definitions for a Benchmark."
method: GET
path: "/v1/benchmarks/{id}/definitions"
tags: ["Benchmark"]
---

# Get scenario definitions for a Benchmark.

`GET /v1/benchmarks/{id}/definitions`

Get scenario definitions for a previously created Benchmark.

## Path parameters

- `id` string, required

## Query parameters

- `limit` integer
- `starting_after` string

## Response `200`

OK

- ScenarioDefinitionListView
  - `scenarios` ScenarioDefinitionView[], required — List of Scenarios matching filter.
    - `id` string, required — The ID of the Scenario.
    - `name` string, required — The name of the Scenario.
    - `environment` ScenarioEnvironment — ScenarioEnvironmentParameters specify the environment in which a Scenario will be run.
      - `blueprint_id` string, nullable — Use the blueprint with matching ID.
      - `snapshot_id` string, nullable — Use the snapshot with matching ID.
      - `launch_parameters` LaunchParameters — LaunchParameters enable you to customize the resources available to your Devbox as well as the environment set up that should be completed before the Devbox is marked as 'running'.
        - `launch_commands` string[], nullable — Set of commands to be run at launch time, before the entrypoint process is run.
        - `resource_size_request` 'X_SMALL' | 'SMALL' | 'MEDIUM' | 'LARGE' | 'X_LARGE' | 'XX_LARGE' | 'CUSTOM_SIZE' — The size of the Devbox resources for Runloop to allocate. X_SMALL: 0.5 cpu x 1GiB memory x 4GiB disk SMALL: 1 cpu x 2GiB memory x 4GiB disk MEDIUM: 2 cpu x 4GiB memory x 8GiB disk LARGE: 2 cpu x 8GiB memory x 16GiB disk X_LARGE: 4 cpu x 16GiB memory x 16GiB disk XX_LARGE: 8 cpu x 32GiB memory x 16GiB disk CUSTOM_SIZE: To choose a custom size, set this enum and also the custom_cpu_cores, custom_gb_memory, and optionally custom_disk_size in launch parameters. CPU must be 0.5, 1, or a multiple of 2 (max 16). Memory must be 1 or a multiple of 2 (max 64GiB). Disk must be a multiple of 2 (min 2GiB, max 64GiB). The cpu:memory ratio must be between 1:2 and 1:8 inclusive.
        - `available_ports` integer[], nullable — [Deprecated] A list of ports to make available on the Devbox. This field is ignored.
        - `keep_alive_time_seconds` integer, nullable — Time in seconds after which Devbox will automatically shutdown. Default is 1 hour. Maximum is 48 hours (172800 seconds).
        - `after_idle` IdleConfigurationParameters
          - `idle_time_seconds` integer, required — After idle_time_seconds, on_idle action will be taken.
          - `on_idle` 'shutdown' | 'suspend', required — Action to take after Devbox idle timer is triggered. shutdown: Shutdown the Devbox. suspend: Suspend the Devbox.
        - `custom_cpu_cores` integer, nullable — Custom CPU cores. Must be 0.5, 1, or a multiple of 2. Max is 16.
        - `custom_gb_memory` integer, nullable — Custom memory size in GiB. Must be 1 or a multiple of 2. Max is 64GiB.
        - `custom_disk_size` integer, nullable — Custom disk size in GiB. Must be a multiple of 2. Min is 2GiB, max is 64GiB.
        - `architecture` 'x86_64' | 'arm64'
        - `user_parameters` UserParameters — Configuration for the Linux user in the Devbox environment.
          - `username` string, required — Username for the Linux user.
          - `uid` integer, required — User ID (UID) for the Linux user. Must be a non-negative integer.
        - `required_services` string[], nullable — A list of ContainerizedService names to be started when a Devbox is created. A valid ContainerizedService must be specified in Blueprint to be started.
        - `network_policy_id` string, nullable — (Optional) ID of the network policy to apply to Devboxes launched with these parameters. When set on a Blueprint launch parameters, Devboxes created from it will inherit this policy unless explicitly overridden.
        - `lifecycle` LifecycleConfigurationParameters — Lifecycle configuration for Devbox idle and resume behavior. Configure idle policy via after_idle and resume triggers via resume_triggers.
          - `after_idle` IdleConfigurationParameters
            - `idle_time_seconds` integer, required — After idle_time_seconds, on_idle action will be taken.
            - `on_idle` 'shutdown' | 'suspend', required — Action to take after Devbox idle timer is triggered. shutdown: Shutdown the Devbox. suspend: Suspend the Devbox.
          - `resume_triggers` ResumeTriggers — Triggers that can resume a suspended Devbox.
            - `http` boolean, nullable — When true, HTTP traffic to a suspended Devbox via tunnel will trigger a resume.
            - `axon_event` boolean, nullable — When true, axon events targeting a suspended Devbox will trigger a resume.
      - `working_directory` string, nullable — The working directory where the agent is expected to fulfill the scenario. Scoring functions also run from the working directory.
    - `input_context` InputContext, required — InputContextView specifies the problem statement along with all additional context for a Scenario.
      - `problem_statement` string, required — The problem statement for the Scenario.
      - `additional_context` object, nullable — Additional JSON structured input context.
    - `scoring_contract` ScoringContract, required — InputContextView specifies the problem statement along with all additional context for a Scenario.
      - `scoring_function_parameters` ScoringFunction[], required — A list of scoring functions used to evaluate the Scenario.
        - `name` string, required — Name of scoring function. Names must only contain [a-zA-Z0-9_-].
        - `scorer` union, required
          - AstGrepScoringFunction — AstGrepScoringFunction utilizes structured coach search for scoring.
            - `lang` string — The language of the pattern.
            - `search_directory` string, required — The path to search.
            - `pattern` string, required — AST pattern to match. Pattern will be passed to ast-grep using the commandline surround by double quotes ("), so make sure to use proper escaping (for example, \$\$\$).
            - `type` 'ast_grep_scorer', required
          - BashScriptScoringFunction — BashScriptScoringFunction is a scoring function specified by a bash script that will be run in the context of your environment.
            - `bash_script` string — A single bash script that sets up the environment, scores, and prints the final score to standard out. Score should be a float between 0.0 and 1.0, and look like "score=[0.0..1.0].
            - `type` 'bash_script_scorer', required
          - CommandScoringFunction — CommandScoringFunction executes a single command and checks the result.The output of the command will be printed. Scoring will passed if the command returns status code 0, otherwise it will be failed.
            - `command` string — The command to execute.
            - `type` 'command_scorer', required
          - CustomScoringFunction — CustomScoringFunction is a custom, user defined scoring function.
            - `custom_scorer_type` string, required — Type of the scoring function, previously registered with Runloop.
            - `scorer_params` object, nullable — Additional JSON structured context to pass to the scoring function.
            - `type` 'custom_scorer', required
          - PythonScriptScoringFunction — PythonScriptScoringFunction will run a python script in the context of your environment as a ScoringFunction.
            - `requirements_contents` string, nullable — Package dependencies to be installed. The requirements should be a valid requirements.txt file.
            - `python_script` string, required — Python script to be run. The script should output the score to standard out as a float between 0.0 and 1.0.
            - `python_version_constraint` string, nullable — Python version to run scoring. Default is "==3.12.10"
            - `type` 'python_script_scorer', required
          - TestBasedScoringFunction — TestBasedScoringFunction writes test files to disk and executes a test command to verify the solution.
            - `test_files` TestFile[] — List of test files to create
              - …
            - `test_command` string — The command to execute for running the tests
            - `type` 'test_based_scorer', required
        - `weight` number, float, required — Weight to apply to scoring function score. Weights of all scoring functions should sum to 1.0.
    - `metadata` object, required — User defined metadata to attach to the scenario for organization.
    - `reference_output` string, nullable — A string representation of the reference output to solve the scenario. Commonly can be the result of a git diff or a sequence of command actions to apply to the environment.
    - `required_environment_variables` string[] — Environment variables required to run the scenario. If any required environment variables are missing, the scenario will fail to start.
    - `required_secret_names` string[] — Environment variables required to run the scenario. If any required secrets are missing, the scenario will fail to start.
    - `is_public` boolean — Whether this scenario is public.
    - `validation_type` 'UNSPECIFIED' | 'FORWARD' | 'REVERSE' | 'EVALUATION'
    - `scorer_timeout_sec` integer, nullable — Timeout for scoring in seconds. Default 30 minutes (1800s).
    - `status` 'active' | 'archived', required
  - `has_more` boolean, required
  - `total_count` integer, nullable

## Changes

- **2026-04-13** `5b536a11a713` — 1 info
  - added the optional property `scenarios/items/environment/launch_parameters/lifecycle/resume_triggers/axon_event` to the response with the `200` status
- **2026-04-09** `a1c7e69cbbf7` — 1 info
  - added the optional property `scenarios/items/environment/launch_parameters/lifecycle` to the response with the `200` status
- **2026-03-24** `32e4b2dfb757` — 1 warning
  - removed the optional property `remaining_count` from the response with the `200` status
- **2026-02-25** `dd0f43e15cb6` — 2 warning
  - added the new `active` enum value to the `scenarios/items/status` response property for the response status `200`
  - added the new `archived` enum value to the `scenarios/items/status` response property for the response status `200`
- **2026-02-19** `8704a652545c` — 4 breaking, 1 info
  - the response property `remaining_count` became nullable for the status `200`
  - the response property `total_count` became nullable for the status `200`
  - the response property `remaining_count` became optional for the status `200`
  - the response property `total_count` became optional for the status `200`
  - …1 more

[Full history](https://skmtc.dev/runloopai/apis/runloop-api/changes/v1/benchmarks/:id/definitions/get.md)

---

[API](https://skmtc.dev/runloopai/apis/runloop-api.md) · [All operations](https://skmtc.dev/runloopai/apis/runloop-api/llms.txt) · [OpenAPI document](https://skmtc-service-production.skmtc.workers.dev/v1/apis/runloopai/runloop-api/revisions/5b536a11a713/schema)