CAcaveman

POST /projects/{projectId}/eval-scenarios

POST /api/v1/projects/{projectId}/eval-scenarios on the Caveman Cloud control plane. Requires any one of the scopes eval:write, scenario:write. Exposed to autonomous agents as the create_scenario action of the caveman_evals_write tool (write access). Author versioned datasets, reusable evaluators and workbenches; create strict suites, ingest evidence, and run baselines. Evidence ingest supports dry_run — always dry-run first. Ask Caveman to draft quality criteria again within the drafting allowance, propose a criterion for a person to turn on, and hold the pull request that adds the Caveman check to CI. Track a criterion on a slice of traces in one step: from an agent it lands proposed, and a person turns it on. Queue units for review and label them (an agent's label is recorded as an agent's, never as human gold), and make a dataset from recorded traffic.

post/projects/{projectId}/eval-scenarios

Request

  • Base URL
    https://api.caveman.so/api/v1
  • URL
    https://api.caveman.so/api/v1/projects/{projectId}/eval-scenarios
  • Auth
    • OAuth 2
      Authorization code: /api/v1/oauth/authorize
      Token URL: /api/v1/oauth/token scopes: eval:write, scenario:write
    • or HTTP bearer
    • or API key in cookie cave_access
    • or API key in cookie __Secure-cave_identity.session_token

Request body

schema_versionunknown required
adapter'python-module-task-v1' | 'node-module-task-v1' required
namestring required
invariantstring required
entrypointstring required
inputunknown required
initial_stateobject required
read_only_toolsstring[]

Declared tool names whose calls change nothing. A trial may then make those calls in another order between two other calls, and skip one unless its recorded answer differs from that same call's latest earlier answer; every other fixture must be called in order. Omit when any tool has side effects you are unsure of.

Response

Canonical immutable dataset scenario version, unexecuted.

dataset_idstring uuid required
version_idstring uuid required
versioninteger required
scenario_sha256string required
source_run_idstring uuid
case_idstring uuid
execution_status'not_executed' | 'queued' | 'running' | 'completed' | 'failed' | 'cancelled' required

Changes

No recorded changes to this operation across all 1 revision of this API.