---
title: "POST /projects/{projectId}/eval-scenarios"
method: POST
path: "/projects/{projectId}/eval-scenarios"
---

# POST /projects/{projectId}/eval-scenarios

`POST /projects/{projectId}/eval-scenarios`

## Request

- Base URL: `https://api.caveman.so/api/v1`
- URL: `https://api.caveman.so/api/v1/projects/{projectId}/eval-scenarios`
- Auth: one of:
  - OAuth 2
    - Authorization code: `/api/v1/oauth/authorize`
    - Token URL: `/api/v1/oauth/token` scopes: `eval:write`, `scenario:write`
  - HTTP bearer
  - API key in cookie `cave_access`
  - API key in cookie `__Secure-cave_identity.session_token`

POST /api/v1/projects/{projectId}/eval-scenarios on the Caveman Cloud control plane. Requires any one of the scopes eval:write, scenario:write. Exposed to autonomous agents as the create_scenario action of the caveman_evals_write tool (write access). Author versioned datasets, reusable evaluators and workbenches; create strict suites, ingest evidence, and run baselines. Evidence ingest supports dry_run — always dry-run first. Ask Caveman to draft quality criteria again within the drafting allowance, propose a criterion for a person to turn on, and hold the pull request that adds the Caveman check to CI. Track a criterion on a slice of traces in one step: from an agent it lands proposed, and a person turns it on. Queue units for review and label them (an agent's label is recorded as an agent's, never as human gold), and make a dataset from recorded traffic.

## Request body

- object
  - `schema_version` unknown, required
  - `adapter` 'python-module-task-v1' | 'node-module-task-v1', required
  - `name` string, required
  - `invariant` string, required
  - `entrypoint` string, required
  - `input` unknown, required
  - `initial_state` object, required
  - `tools` object[], required
    - `name` string, required
    - `arguments` unknown, required
    - `result` unknown, required
    - `error` string
  - `read_only_tools` string[]: Declared tool names whose calls change nothing. A trial may then make those calls in another order between two other calls, and skip one unless its recorded answer differs from that same call's latest earlier answer; every other fixture must be called in order. Omit when any tool has side effects you are unsure of.
  - `assertions` object[], required
    - `subject` 'output' | 'state', required
    - `pointer` string, required
    - `operator` 'equals' | 'contains' | 'not_contains', required
    - `value` unknown, required
  - `expected_failure` object
    - `exception_type` string, required
    - `path` string, required
    - `function` string, required
  - `historical` object[], required
    - `commit_sha` string, required
    - `expected` 'passed' | 'failed', required
  - `provenance` object, required
    - `evidence_digests` string[], required
    - `fixture_basis` 'synthetic', required
    - `fixture_rationale` string, required

## Response `201`

Canonical immutable dataset scenario version, unexecuted.

- AgentTaskScenarioVersion
  - `dataset_id` string, uuid, required
  - `version_id` string, uuid, required
  - `version` integer, required
  - `definition` AgentTaskScenario, required
    - `schema_version` unknown, required
    - `adapter` 'python-module-task-v1' | 'node-module-task-v1', required
    - `name` string, required
    - `invariant` string, required
    - `entrypoint` string, required
    - `input` unknown, required
    - `initial_state` object, required
    - `tools` object[], required
      - `name` string, required
      - `arguments` unknown, required
      - `result` unknown, required
      - `error` string
    - `read_only_tools` string[]: Declared tool names whose calls change nothing: a trial may make them in another order between two other calls, and skip one whose answer did not change since the same call was last made.
    - `assertions` object[], required
      - `subject` 'output' | 'state', required
      - `pointer` string, required
      - `operator` 'equals' | 'contains' | 'not_contains', required
      - `value` unknown, required
    - `expected_failure` object
      - `exception_type` string, required
      - `path` string, required
      - `function` string, required
    - `historical` object[], required
      - `commit_sha` string, required
      - `expected` 'passed' | 'failed', required
    - `provenance` object, required
      - `evidence_digests` string[], required
      - `fixture_basis` 'synthetic', required
      - `fixture_rationale` string, required
  - `scenario_sha256` string, required
  - `source_run_id` string, uuid
  - `case_id` string, uuid
  - `execution_status` 'not_executed' | 'queued' | 'running' | 'completed' | 'failed' | 'cancelled', required
  - `executions` object[], required
    - `run_id` string, uuid, required
    - `workbench_id` string, uuid, required
    - `native_run_id` string, uuid, required
    - `commit_sha` string, required
    - `status` 'queued' | 'running' | 'completed' | 'failed' | 'cancelled', required
    - `assessment` unknown, required
    - `created_at` string, date-time, required

## Other responses

- `422`: Unsupported task definition.
- `4XX`: Structured failure. `error.code` is stable; `error.request_id` is echoed in the x-cave-request-id header.
- `5XX`: Structured failure. `error.code` is stable; `error.request_id` is echoed in the x-cave-request-id header.

---

[API](https://skmtc.dev/caveman/apis/caveman-cloud-api.md) - [All operations](https://skmtc.dev/caveman/apis/caveman-cloud-api/llms.txt) - [OpenAPI document](https://skmtc.dev/caveman/apis/caveman-cloud-api/revisions/508312d01cfc?raw)