Evaluations

Start an eval run

Runs the eval against its dataset, creating one real agent generation per item and scoring the outputs.

wait: true executes the run synchronously and returns it terminal, with its scores. The dataset is capped at 25 items for a synchronous run; a larger one is rejected with 400 rather than partially scored.

wait: false (the default) enqueues one task per item and returns immediately with status: "queued". A worker executes the items and the run settles itself; poll GET /evals/{eval_id}/runs/{eval_run_id} for the terminal status. There is no item cap on a queued run.

The whole run is pinned to one agent version, stamped on agent_version: pass one explicitly to evaluate a canary before promoting it, or omit it to use the active release's stable version (or the live draft when no release is in effect). Without the pin, release assignment would bucket each item independently and blend two configs into a single score.

With baseline_run_id, the finished run's aggregate_scores.baseline carries per-scorer deltas against that run, computed over the items present and scorable in both runs, with the divergence counted. A delta over a shifted dataset is therefore never presented as a clean comparison.

post/v1/projects/{project_id}/evals/{eval_id}/runs

Request

  • Base URL: https://api.naturali.ai
  • URL: https://api.naturali.ai/v1/projects/{project_id}/evals/{eval_id}/runs
  • Auth: one of:
    • HTTP bearer
    • OAuth 2 (authorization code, authorization URL https://api.naturali.ai/authorize, token URL https://api.naturali.ai/token, scopes: mcp:access)

Path parameters

eval_idstring required
Example:eval_V1StGXR8Z5jdHi6B

Eval ID

Request body

waitboolean

True runs the eval synchronously (25-item cap) and returns a terminal run with its scores. False — the default — enqueues the items and returns a queued run immediately.

agent_versioninteger nullable

An archived agent version to evaluate. Defaults to the active release's stable version, or the live draft version when no release is in effect.

baseline_run_idstring nullable

A terminal run of the same eval to compare against. The finished run's aggregate_scores.baseline reports per-scorer deltas over the item intersection. A run of a different eval is rejected with 400.

metadataMetadataBag

Caller-owned annotations on a resource, stored as the object they were written as: the types a value was written with are the types a read returns, so a filter can ask an ordering question about a number. Unlike other body fields, keys are stored and returned verbatim in the casing supplied — they are not converted between snake_case and camelCase.

No key is reserved, and that is the point: every piece of state the platform owns lives in its own typed column, so nothing written here reaches platform state. The platform never reads the bag — it is not an IAM context, not a policy input and not part of a prompt — which is what separates it from a tag bag.

tool_contextobject

Key/value context forwarded to every item's generation, so an agent whose tools authorize through tool_context is scored against the configuration it runs in production rather than with an empty bag. Each key is forwarded as one X-Naturali-Context-<key> header and resolves any {{context:<key>}} token in a bound tool's headers or preset_parameters.

Stored on the run and re-read per item, since a queued run (the default) is driven by a worker with no request behind it. Write-only: no read of the run returns it, unlike metadata — a run is a report other people read, and a credential in it is not theirs to see. Cleared once the run reaches a terminal state.

An eval generation has no session, so the reserved keys session_id, actor_id and actor_external_id are dropped (in any casing) rather than forwarded. Every other key becomes an HTTP header name and must match that grammar, or the request is rejected with 400 INVALID_TOOL_CONTEXT_KEY and no run is created.

Example request

{
  "wait": true,
  "agent_version": 3,
  "baseline_run_id": "evrun_V1StGXR8Z5jdHi6B",
  "metadata": {
    "author": "John",
    "revision": 2
  },
  "tool_context": {
    "ocaToken": "eyJhbGciOiJIUzI1NiJ9.abc",
    "tenant": "acme"
  }
}

Response

Eval run finished (wait: true) or queued (wait: false)

idstring
eval_idstring
agent_versioninteger

The one agent version every item in this run executed against

decider_versionsobject nullable

The decider version each decider scorer grades under, keyed by scorer name and resolved when the run started. Null when the eval has no decider scorer.

status'queued' | 'running' | 'completed' | 'failed' | 'cancelled'
baseline_run_idstring nullable
trigger_idstring nullable

The trigger that started this run — set when a schedule (or a manual trigger fire) started it, null for a run started through this API. Kept even if the trigger is later deleted.

passedboolean nullable

Null when the eval declares no pass_threshold, and until the run is terminal

item_countinteger
completed_countinteger
errored_countinteger
metadataMetadataBag

Caller-owned annotations on a resource, stored as the object they were written as: the types a value was written with are the types a read returns, so a filter can ask an ordering question about a number. Unlike other body fields, keys are stored and returned verbatim in the casing supplied — they are not converted between snake_case and camelCase.

No key is reserved, and that is the point: every piece of state the platform owns lives in its own typed column, so nothing written here reaches platform state. The platform never reads the bag — it is not an IAM context, not a policy input and not part of a prompt — which is what separates it from a tag bag.

started_atstring date-time nullable
finished_atstring date-time nullable
created_atstring date-time

Example response

{
  "id": "evrun_V1StGXR8Z5jdHi6B",
  "eval_id": "eval_V1StGXR8Z5jdHi6B",
  "agent_version": 3,
  "decider_versions": {
    "review": 2
  },
  "baseline_run_id": "evrun_V1StGXR8Z5jdHi6B",
  "trigger_id": "trg_V1StGXR8Z5jdHi6B",
  "aggregate_scores": {
    "pass_rate_interval": {
      "low": 0.63,
      "high": 0.84,
      "level": 0.95
    },
    "baseline": {
      "run_id": "evrun_V1StGXR8Z5jdHi6B",
      "p_value": 0.03,
      "grouping": {
        "group_by": "kind"
      }
    },
    "grouping": {
      "group_by": "kind"
    }
  },
  "metadata": {
    "author": "John",
    "revision": 2
  }
}

Changes

No recorded changes to this operation across all 1 revision of this API.