CAcaveman

POST /projects/{projectId}/criteria

POST /api/v1/projects/{projectId}/criteria on the Caveman Cloud control plane. Requires the eval:write scope. Exposed to autonomous agents as the create_criterion action of the caveman_evals_write tool (write access). Author versioned datasets, reusable evaluators and workbenches; create strict suites, ingest evidence, and run baselines. Evidence ingest supports dry_run — always dry-run first. Ask Caveman to draft quality criteria again within the drafting allowance, propose a criterion for a person to turn on, and hold the pull request that adds the Caveman check to CI. Track a criterion on a slice of traces in one step: from an agent it lands proposed, and a person turns it on. Queue units for review and label them (an agent's label is recorded as an agent's, never as human gold), and make a dataset from recorded traffic.

post/projects/{projectId}/criteria

Request

  • Base URL
    https://api.caveman.so/api/v1
  • URL
    https://api.caveman.so/api/v1/projects/{projectId}/criteria
  • Auth
    • OAuth 2
      Authorization code: /api/v1/oauth/authorize
      Token URL: /api/v1/oauth/token scopes: eval:write
    • or HTTP bearer
    • or API key in cookie cave_access
    • or API key in cookie __Secure-cave_identity.session_token

Request body

namestring required
kind'question' | 'output_grader' | 'assertion' | 'code' required
scope'span' | 'trace' | 'task' | 'row' required
runtime'python' | 'javascript' | 'typescript'
sourcestring
min_scorenumber
workload_idstring uuid nullable
statementstring
pass_defstring
fail_defstring
na_defstring
answer'pass_fail' | 'score' | 'category' | 'pairwise' | 'answer_match'
categoriesstring[]
grader_typestring
optionsobject
input_mappingobject
gatingboolean

Response

Successful response. Exact status depends on synchronous or durable delivery outcome.

Changes

No recorded changes to this operation across all 1 revision of this API.