Scenario

Create a Scenario.

Changed on

Create a Scenario, a repeatable AI coding evaluation test that defines the starting environment as well as evaluation success criteria.

post/v1/scenarios

Request

  • Base URL: https://api.runloop.ai
  • URL: https://api.runloop.ai/v1/scenarios
  • Auth: HTTP bearer

Request body

namestring required

Name of the scenario.

metadataobject nullable

User defined metadata to attach to the scenario for organization.

reference_outputstring nullable

A string representation of the reference output to solve the scenario. Commonly can be the result of a git diff or a sequence of command actions to apply to the environment.

required_environment_variablesstring[] nullable

Environment variables required to run the scenario. If these variables are not provided, the scenario will fail to start.

required_secret_namesstring[] nullable

Secrets required to run the scenario (user secret name to scenario required secret name). If these secrets are not provided or the mapping is incorrect, the scenario will fail to start.

validation_type'UNSPECIFIED' | 'FORWARD' | 'REVERSE' | 'EVALUATION'
scorer_timeout_secinteger nullable

Timeout for scoring in seconds. Default 30 minutes (1800s).

Response

OK

idstring required

The ID of the Scenario.

namestring required

The name of the Scenario.

metadataobject required

User defined metadata to attach to the scenario for organization.

reference_outputstring nullable

A string representation of the reference output to solve the scenario. Commonly can be the result of a git diff or a sequence of command actions to apply to the environment.

required_environment_variablesstring[]

Environment variables required to run the scenario. If any required environment variables are missing, the scenario will fail to start.

required_secret_namesstring[]

Environment variables required to run the scenario. If any required secrets are missing, the scenario will fail to start.

is_publicboolean

Whether this scenario is public.

validation_type'UNSPECIFIED' | 'FORWARD' | 'REVERSE' | 'EVALUATION'
scorer_timeout_secinteger nullable

Timeout for scoring in seconds. Default 30 minutes (1800s).

status'active' | 'archived' required

Changes

  • b4ffb9a8aff3Revision changes
    • info

      added the new optional request property //////////

    • info

      added the optional property ////////// to the response with the status

  • d3ddb452754cRevision changes
    • breaking

      the request property became not nullable

    • breaking

      the request property / became not nullable

    • breaking

      the request property became not nullable

    • breaking

      the request property became not nullable

    • breaking

      the request property became not nullable

    • breaking

      the request property became not nullable

    • breaking

      the request property became not nullable

    • breaking

      the request property //////// became not nullable

    • breaking

      the request property //////// became not nullable

    • breaking

      the request property //////// became not nullable

    • breaking

      the request property became not nullable

    • breaking

      removed the enum value EVALUATION of the request property

    • breaking

      removed the enum value FORWARD of the request property

    • breaking

      removed the enum value REVERSE of the request property

    • breaking

      removed the enum value UNSPECIFIED of the request property

    • breaking

      the request property type changed from object to no type

    • breaking

      the request property type changed from string to no type

    • breaking

      the response's property type changed from object to no type for status

    • breaking

      the response's property type changed from string to no type for status

    • warning

      removed the request property /

    • warning

      removed the request property /

    • warning

      removed the request property /

    • warning

      removed the request property /

    • warning

      removed the optional property / from the response with the status

    • warning

      removed the optional property / from the response with the status

    • warning

      removed the optional property / from the response with the status

    • warning

      removed the optional property / from the response with the status

    • info

      added subschema #2 to the request property anyOf list

    • info

      added subschema #2 to the request property anyOf list

    • info

      added subschema #2 to the response property anyOf list for the response status

    • info

      added subschema #2 to the response property anyOf list for the response status

    • info

      removed the EVALUATION enum value from the response property for the response status

    • info

      removed the FORWARD enum value from the response property for the response status

    • info

      removed the REVERSE enum value from the response property for the response status

    • info

      removed the UNSPECIFIED enum value from the response property for the response status