---
title: "Start an eval run"
method: POST
path: "/v1/projects/{project_id}/evals/{eval_id}/runs"
tags: ["Evaluations"]
---

# Start an eval run

`POST /v1/projects/{project_id}/evals/{eval_id}/runs`

## Request

- Base URL: `https://api.naturali.ai`
- URL: `https://api.naturali.ai/v1/projects/{project_id}/evals/{eval_id}/runs`
- Auth: one of:
  - HTTP bearer
  - OAuth 2 (authorization code, authorization URL `https://api.naturali.ai/authorize`, token URL `https://api.naturali.ai/token`, scopes: `mcp:access`)

Runs the eval against its dataset, creating one real agent generation per item and scoring the outputs.

`wait: true` executes the run synchronously and returns it terminal, with its scores. The dataset is capped at 25 items for a synchronous run; a larger one is rejected with 400 rather than partially scored.

`wait: false` (the default) enqueues one task per item and returns immediately with `status: "queued"`. A worker executes the items and the run settles itself; poll `GET /evals/{eval_id}/runs/{eval_run_id}` for the terminal status. There is no item cap on a queued run.

The whole run is pinned to **one** agent version, stamped on `agent_version`: pass one explicitly to evaluate a canary before promoting it, or omit it to use the active release's stable version (or the live draft when no release is in effect). Without the pin, release assignment would bucket each item independently and blend two configs into a single score.

With `baseline_run_id`, the finished run's `aggregate_scores.baseline` carries per-scorer deltas against that run, computed over the items present and scorable in **both** runs, with the divergence counted. A delta over a shifted dataset is therefore never presented as a clean comparison.

## Path parameters

- `eval_id` string, required

## Request body

- object
  - `wait` boolean: True runs the eval synchronously (25-item cap) and returns a terminal run with its scores. False — the default — enqueues the items and returns a `queued` run immediately.
  - `agent_version` integer, nullable: An archived agent version to evaluate. Defaults to the active release's stable version, or the live draft version when no release is in effect.
  - `baseline_run_id` string, nullable: A terminal run of the same eval to compare against. The finished run's `aggregate_scores.baseline` reports per-scorer deltas over the item intersection. A run of a different eval is rejected with 400.
  - `metadata` MetadataBag: Caller-owned annotations on a resource, stored as the object they were written as: the types a value was written with are the types a read returns, so a filter can ask an ordering question about a number. Unlike other body fields, keys are stored and returned verbatim in the casing supplied — they are not converted between snake_case and camelCase. No key is reserved, and that is the point: every piece of state the platform owns lives in its own typed column, so nothing written here reaches platform state. The platform never reads the bag — it is not an IAM context, not a policy input and not part of a prompt — which is what separates it from a tag bag.
  - `tool_context` object: Key/value context forwarded to every item's generation, so an agent whose tools authorize through `tool_context` is scored against the configuration it runs in production rather than with an empty bag. Each key is forwarded as one `X-Naturali-Context-<key>` header and resolves any `{{context:<key>}}` token in a bound tool's headers or `preset_parameters`. Stored on the run and re-read per item, since a queued run (the default) is driven by a worker with no request behind it. **Write-only**: no read of the run returns it, unlike `metadata` — a run is a report other people read, and a credential in it is not theirs to see. Cleared once the run reaches a terminal state. An eval generation has no session, so the reserved keys `session_id`, `actor_id` and `actor_external_id` are dropped (in any casing) rather than forwarded. Every other key becomes an HTTP header name and must match that grammar, or the request is rejected with `400 INVALID_TOOL_CONTEXT_KEY` and no run is created.

## Response `201`

Eval run finished (`wait: true`) or queued (`wait: false`)

- EvalRun
  - `id` string
  - `eval_id` string
  - `agent_version` integer: The one agent version every item in this run executed against
  - `decider_versions` object, nullable: The decider version each `decider` scorer grades under, keyed by scorer name and resolved when the run started. Null when the eval has no decider scorer.
  - `status` 'queued' | 'running' | 'completed' | 'failed' | 'cancelled'
  - `baseline_run_id` string, nullable
  - `trigger_id` string, nullable: The trigger that started this run — set when a schedule (or a manual trigger fire) started it, null for a run started through this API. Kept even if the trigger is later deleted.
  - `aggregate_scores` AggregateScores, nullable: Per-scorer rollup plus the run-level pass rate; null until the run is terminal
    - `scorers` object
    - `pass_rate` number, nullable: Passed items over non-errored items; null when nothing was scorable
    - `pass_rate_interval` PassRateInterval, nullable: 95% Wilson score interval around `pass_rate`: the range the pass rate of this agent over items like these plausibly spans, given how many items were scored. Null exactly when `pass_rate` is.
      - `low` number
      - `high` number
      - `level` number: The confidence level, always 0.95
    - `scored_item_count` integer: Items that produced a score — errored items are excluded
    - `baseline` BaselineComparison, nullable: Comparison against the run named by `baseline_run_id`; absent when the run named none. Every number here is computed over the **item intersection** — the dataset items present and scorable in both runs — because a delta only means something when both sides answered the same question. The compared/added/removed counts make any dataset drift visible instead of letting it read as agent regression. Positive deltas mean this run scored higher than the baseline.
      - `run_id` string
      - `compared_item_count` integer: Items scorable in both runs — the basis of every delta
      - `added_item_count` integer: Scorable here but not in the baseline (added, or errored there)
      - `removed_item_count` integer: Scorable in the baseline but not here (removed, or errored here)
      - `pass_rate_delta` number, nullable: Run-level pass-rate delta over the intersection; null when the two runs share no comparable item
      - `flipped` FlippedItems: Compared items that changed verdict. `pass_rate_delta` is `(improved - regressed) / compared_item_count`.
        - `improved` integer: Passed in this run, failed in the baseline
        - `regressed` integer: Failed in this run, passed in the baseline
      - `p_value` number, nullable: Exact McNemar p-value of the `flipped` split: the probability of a split at least this uneven were an item equally likely to flip either way, which is what an unchanged agent produces. `1` when no item flipped; null when the runs share no comparable item.
      - `scorers` object: Per-scorer deltas, keyed by scorer type. A scorer only one of the two runs ran is omitted rather than compared against nothing.
      - `grouping` BaselineGrouping: The baseline comparison per group; absent when the eval declares no `group_by`. Both runs' items are grouped by the same labels, so an item relabelled between the runs is compared within one group.
        - `group_by` string
        - `groups` object
        - `ungrouped_item_count` integer: Items scorable in both runs that name no group
    - `grouping` RunGrouping: Scores rolled up per value of the eval's `group_by` key; absent when the eval declares none. An item belongs to the group its `metadata` names under that key, read when the run settles; only a string names a group. Errored items count in no group.
      - `group_by` string: The metadata key the groups were read from
      - `groups` object: Group value → that group's figures
      - `ungrouped_item_count` integer: Scored items whose `metadata` holds no string under `group_by`, or whose dataset item was deleted
  - `passed` boolean, nullable: Null when the eval declares no pass_threshold, and until the run is terminal
  - `item_count` integer
  - `completed_count` integer
  - `errored_count` integer
  - `metadata` MetadataBag: Caller-owned annotations on a resource, stored as the object they were written as: the types a value was written with are the types a read returns, so a filter can ask an ordering question about a number. Unlike other body fields, keys are stored and returned verbatim in the casing supplied — they are not converted between snake_case and camelCase. No key is reserved, and that is the point: every piece of state the platform owns lives in its own typed column, so nothing written here reaches platform state. The platform never reads the bag — it is not an IAM context, not a policy input and not part of a prompt — which is what separates it from a tag bag.
  - `started_at` string, date-time, nullable
  - `finished_at` string, date-time, nullable
  - `created_at` string, date-time

## Other responses

- `400`: Bad request (non-boolean wait, dataset empty or over the synchronous cap, unknown agent_version, invalid baseline, scorers no longer valid against the agent, a `tool_context` key that cannot become a header)
- `401`: Unauthorized
- `403`: Forbidden
- `404`: Eval not found
- `500`: Internal server error

---

[API](https://skmtc.dev/naturali/apis/naturali-ai-api.md) · [All operations](https://skmtc.dev/naturali/apis/naturali-ai-api/llms.txt) · [OpenAPI document](https://skmtc.dev/naturali/apis/naturali-ai-api/revisions/e4905b78f422?raw)