---
title: "Get scenario definitions for a Benchmark."
method: GET
path: "/v1/benchmarks/{id}/definitions"
tags: ["Benchmark"]
---

# Get scenario definitions for a Benchmark.

`GET /v1/benchmarks/{id}/definitions`

Get scenario definitions for a previously created Benchmark.

## Path parameters

- `id` string, required

## Query parameters

- `limit` integer
- `starting_after` string

## Response `200`

OK

- ScenarioDefinitionListView
  - `scenarios` ScenarioDefinitionView[], required — List of Scenarios matching filter.
    - `id` string, required — The ID of the Scenario.
    - `name` string, required — The name of the Scenario.
    - `environment` ScenarioEnvironment — ScenarioEnvironmentParameters specify the environment in which a Scenario will be run.
      - `blueprint_id` string, nullable — Use the blueprint with matching ID.
      - `snapshot_id` string, nullable — Use the snapshot with matching ID.
      - `launch_parameters` LaunchParameters — LaunchParameters enable you to customize the resources available to your Devbox as well as the environment set up that should be completed before the Devbox is marked as 'running'.
        - `launch_commands` string[], nullable — Set of commands to be run at launch time, before the entrypoint process is run.
        - `resource_size_request` 'X_SMALL' | 'SMALL' | 'MEDIUM' | 'LARGE' | 'X_LARGE' | 'XX_LARGE' | 'CUSTOM_SIZE' — The size of the Devbox resources for Runloop to allocate. X_SMALL: 0.5 cpu x 1GiB memory x 4GiB disk SMALL: 1 cpu x 2GiB memory x 4GiB disk MEDIUM: 2 cpu x 4GiB memory x 8GiB disk LARGE: 2 cpu x 8GiB memory x 16GiB disk X_LARGE: 4 cpu x 16GiB memory x 16GiB disk XX_LARGE: 8 cpu x 32GiB memory x 16GiB disk CUSTOM_SIZE: To choose a custom size, set this enum and also the custom_cpu_cores, custom_gb_memory, and optionally custom_disk_size in launch parameters. CPU must be 0.5, 1, or a multiple of 2 (max 16). Memory must be 1 or a multiple of 2 (max 64GiB). Disk must be a multiple of 2 (min 2GiB, max 64GiB). The cpu:memory ratio must be between 1:2 and 1:8 inclusive.
        - `keep_alive_time_seconds` integer, nullable — Time in seconds after which Devbox will automatically shutdown. Default is 1 hour. Maximum is 48 hours (172800 seconds).
        - `available_ports` integer[], nullable — A list of ports to make available on the Devbox. Only ports made available will be surfaced to create tunnels via the 'createTunnel' API.
        - `after_idle` IdleConfigurationParameters
          - `idle_time_seconds` integer, required — After idle_time_seconds, on_idle action will be taken.
          - `on_idle` 'shutdown' | 'suspend', required — Action to take after Devbox idle timer is triggered. shutdown: Shutdown the Devbox. suspend: Suspend the Devbox.
        - `custom_cpu_cores` integer, nullable — Custom CPU cores. Must be 0.5, 1, or a multiple of 2. Max is 16.
        - `custom_gb_memory` integer, nullable — Custom memory size in GiB. Must be 1 or a multiple of 2. Max is 64GiB.
        - `custom_disk_size` integer, nullable — Custom disk size in GiB. Must be a multiple of 2. Min is 2GiB, max is 64GiB.
        - `architecture` 'x86_64' | 'arm64'
        - `user_parameters` UserParameters — Configuration for the Linux user in the Devbox environment.
          - `username` string, required — Username for the Linux user.
          - `uid` integer, required — User ID (UID) for the Linux user. Must be a non-negative integer.
        - `required_services` string[], nullable — A list of ContainerizedService names to be started when a Devbox is created. A valid ContainerizedService must be specified in Blueprint to be started.
        - `network_policy_id` string, nullable — (Optional) ID of the network policy to apply to Devboxes launched with these parameters. When set on a Blueprint launch parameters, Devboxes created from it will inherit this policy unless explicitly overridden.
      - `working_directory` string, nullable — The working directory where the agent is expected to fulfill the scenario. Scoring functions also run from the working directory.
    - `input_context` InputContext, required — InputContextView specifies the problem statement along with all additional context for a Scenario.
      - `problem_statement` string, required — The problem statement for the Scenario.
      - `additional_context` object, nullable — Additional JSON structured input context.
    - `scoring_contract` ScoringContract, required — InputContextView specifies the problem statement along with all additional context for a Scenario.
      - `scoring_function_parameters` ScoringFunction[], required — A list of scoring functions used to evaluate the Scenario.
        - `name` string, required — Name of scoring function. Names must only contain [a-zA-Z0-9_-].
        - `scorer` union, required
          - AstGrepScoringFunction — AstGrepScoringFunction utilizes structured coach search for scoring.
            - `lang` string — The language of the pattern.
            - `search_directory` string, required — The path to search.
            - `pattern` string, required — AST pattern to match. Pattern will be passed to ast-grep using the commandline surround by double quotes ("), so make sure to use proper escaping (for example, \$\$\$).
            - `type` 'ast_grep_scorer', required
          - BashScriptScoringFunction — BashScriptScoringFunction is a scoring function specified by a bash script that will be run in the context of your environment.
            - `bash_script` string — A single bash script that sets up the environment, scores, and prints the final score to standard out. Score should be a float between 0.0 and 1.0, and look like "score=[0.0..1.0].
            - `type` 'bash_script_scorer', required
          - CommandScoringFunction — CommandScoringFunction executes a single command and checks the result.The output of the command will be printed. Scoring will passed if the command returns status code 0, otherwise it will be failed.
            - `command` string — The command to execute.
            - `type` 'command_scorer', required
          - CustomScoringFunction — CustomScoringFunction is a custom, user defined scoring function.
            - `custom_scorer_type` string, required — Type of the scoring function, previously registered with Runloop.
            - `scorer_params` object, nullable — Additional JSON structured context to pass to the scoring function.
            - `type` 'custom_scorer', required
          - PythonScriptScoringFunction — PythonScriptScoringFunction will run a python script in the context of your environment as a ScoringFunction.
            - `requirements_contents` string, nullable — Package dependencies to be installed. The requirements should be a valid requirements.txt file.
            - `python_script` string, required — Python script to be run. The script should output the score to standard out as a float between 0.0 and 1.0.
            - `python_version_constraint` string, nullable — Python version to run scoring. Default is "==3.12.10"
            - `type` 'python_script_scorer', required
          - TestBasedScoringFunction — TestBasedScoringFunction writes test files to disk and executes a test command to verify the solution.
            - `test_files` TestFile[] — List of test files to create
              - …
            - `test_command` string — The command to execute for running the tests
            - `type` 'test_based_scorer', required
        - `weight` number, float, required — Weight to apply to scoring function score. Weights of all scoring functions should sum to 1.0.
    - `metadata` object, required — User defined metadata to attach to the scenario for organization.
    - `reference_output` string, nullable — A string representation of the reference output to solve the scenario. Commonly can be the result of a git diff or a sequence of command actions to apply to the environment.
    - `required_environment_variables` string[] — Environment variables required to run the scenario. If any required environment variables are missing, the scenario will fail to start.
    - `required_secret_names` string[] — Environment variables required to run the scenario. If any required secrets are missing, the scenario will fail to start.
    - `is_public` boolean — Whether this scenario is public.
    - `validation_type` 'UNSPECIFIED' | 'FORWARD' | 'REVERSE' | 'EVALUATION'
    - `scorer_timeout_sec` integer, nullable — Timeout for scoring in seconds. Default 30 minutes (1800s).
  - `has_more` boolean, required
  - `total_count` integer, required
  - `remaining_count` integer, required

## Changes

- **2026-02-12** `c8d61a0c8b88` — 1 info
  - added the optional property `scenarios/items/scorer_timeout_sec` to the response with the `200` status
- **2026-01-19** `8ebd9dbfacb1` — 1 info
  - added the optional property `scenarios/items/environment/launch_parameters/network_policy_id` to the response with the `200` status
- **2025-09-30** `9b2e136aedff` — 1 warning
  - added the new `UNSPECIFIED` enum value to the `scenarios/items/validation_type` response property for the response status `200`

[Change history](https://skmtc.dev/runloopai/apis/runloop-api/changes/v1/benchmarks/:id/definitions/get.md)

---

[API](https://skmtc.dev/runloopai/apis/runloop-api.md) · [All operations](https://skmtc.dev/runloopai/apis/runloop-api/llms.txt) · [OpenAPI document](https://skmtc-service-production.skmtc.workers.dev/v1/apis/runloopai/runloop-api/revisions/c8d61a0c8b88/schema)