---
title: "POST /projects/{projectId}/criteria/test"
method: POST
path: "/projects/{projectId}/criteria/test"
---

# POST /projects/{projectId}/criteria/test

`POST /projects/{projectId}/criteria/test`

## Request

- Base URL: `https://api.caveman.so/api/v1`
- URL: `https://api.caveman.so/api/v1/projects/{projectId}/criteria/test`
- Auth: one of:
  - OAuth 2
    - Authorization code: `/api/v1/oauth/authorize`
    - Token URL: `/api/v1/oauth/token` scopes: `eval:write`
  - HTTP bearer
  - API key in cookie `cave_access`
  - API key in cookie `__Secure-cave_identity.session_token`

POST /api/v1/projects/{projectId}/criteria/test on the Caveman Cloud control plane. Requires the eval:write scope. Exposed to autonomous agents as the test_criterion action of the caveman_evals_write tool (write access). Author versioned datasets, reusable evaluators and workbenches; create strict suites, ingest evidence, and run baselines. Evidence ingest supports dry_run — always dry-run first. Ask Caveman to draft quality criteria again within the drafting allowance, propose a criterion for a person to turn on, and hold the pull request that adds the Caveman check to CI. Track a criterion on a slice of traces in one step: from an agent it lands proposed, and a person turns it on. Queue units for review and label them (an agent's label is recorded as an agent's, never as human gold), and make a dataset from recorded traffic.

## Request body

- object
  - `criterion` union, required
    - object
      - `name` string
      - `kind` 'question' | 'output_grader' | 'assertion' | 'code', required
      - `scope` 'span' | 'trace' | 'task' | 'row', required
      - `runtime` 'python' | 'javascript' | 'typescript'
      - `source` string
      - `min_score` number
      - `judge` object, nullable
        - `connection` string, required
        - `model_id` string
        - `temperature` number
      - `workload_id` string, uuid, nullable
      - `statement` string
      - `pass_def` string
      - `fail_def` string
      - `na_def` string
      - `answer` 'pass_fail' | 'score' | 'category' | 'pairwise' | 'answer_match'
      - `categories` string[]
      - `grader_type` string
      - `options` object
      - `input_mapping` object
      - `gating` boolean
    - object
      - `criterion_id` string, uuid, required
      - `version` integer
  - `on` union, required
    - object
      - `kind` 'recent', required
      - `scope` 'trace' | 'task', required
      - `workload_id` string, uuid, nullable
      - `window_hours` integer
      - `filter` union[]
        - union
          - object
            - `field` string, required
            - `op` 'is' | 'is_not' | 'in' | 'not_in' | 'contains' | 'not_contains' | 'starts_with' | 'regex' | 'gt' | 'gte' | 'lt' | 'lte' | 'exists' | 'not_exists'
            - `rule` string: Older name of op; matches_regex means regex. Send op or rule, never both.
            - `value` union
              - …
          - object
            - `all` union[], required
              - …
          - object
            - `any` union[], required
              - …
      - `limit` integer
    - object
      - `kind` 'dataset', required
      - `dataset_id` string, uuid, required
      - `limit` integer
    - object
      - `kind` 'inline', required
      - `input` unknown
      - `output` unknown, required
      - `expected` unknown
      - `baseline` unknown
      - `metadata` object

## Response `2XX`

Successful response. Exact status depends on synchronous or durable delivery outcome.

## Other responses

- `4XX`: Structured failure. `error.code` is stable; `error.request_id` is echoed in the x-cave-request-id header.
- `5XX`: Structured failure. `error.code` is stable; `error.request_id` is echoed in the x-cave-request-id header.

---

[API](https://skmtc.dev/caveman/apis/caveman-cloud-api.md) - [All operations](https://skmtc.dev/caveman/apis/caveman-cloud-api/llms.txt) - [OpenAPI document](https://skmtc.dev/caveman/apis/caveman-cloud-api/revisions/508312d01cfc?raw)