---
title: "Create an eval"
method: POST
path: "/v1/projects/{project_id}/evals"
tags: ["Evaluations"]
---

# Create an eval

`POST /v1/projects/{project_id}/evals`

## Request

- Base URL: `https://api.naturali.ai`
- URL: `https://api.naturali.ai/v1/projects/{project_id}/evals`
- Auth: one of:
  - HTTP bearer
  - OAuth 2 (authorization code, authorization URL `https://api.naturali.ai/authorize`, token URL `https://api.naturali.ai/token`, scopes: `mcp:access`)

Binds an agent under test to a dataset and a list of scorers. The agent and the dataset must belong to the same project as the eval; a cross-project reference is rejected with 400.

Scorer config is frozen here rather than read from the agent at run time, so two runs of the same eval are always judged by the same criteria and their comparison measures the agent instead of the config drifting underneath it. Each scorer `type` may appear at most once.

## Request body

- object
  - `name` string, required: Unique name within the project
  - `agent_id` string, required: The agent under test
  - `dataset_id` string, required: The dataset to run it against
  - `scorers` union[], required: Scorer configs, a discriminated union on `type`. Each type may appear at most once. Every scorer produces `{ score: 0–1, passed: boolean }`; binary scorers emit 0 or 1. `exact_match` compares the trimmed output text to `expected_output`. `contains` looks for `value` in the output text. `json_logic` evaluates `expression` over `{ input, output, object, expected, item.metadata }`, where `object` is the structured output (absent when the agent has no `output_schema`). `output_schema` validates the structured output against the scorer's own `schema`, falling back to the agent's; it requires the agent to carry an `output_schema`, because without one the platform emits no structured output and every item would score 0. `llm_judge` grades the output with a model completion, returning a continuous score plus its `reasoning`. Its `pass_threshold` is required: a continuous score says nothing about where "good enough" is, and a defaulted cutoff would silently decide the gate. `embedding_similarity` embeds the output text and `expected_output` with the platform's configured embedding model (`EMBEDDING_PROVIDER` / `EMBEDDING_MODEL` — the same stack document ingestion uses) and scores their cosine similarity, clamped to 0-1. Its `pass_threshold` is required for the same reason as the judge's. An item without an `expected_output` scores 0; an embedding backend failure marks the **item** errored, never a score of 0. `tool` runs a custom scoring algorithm: a server-callable project tool the engine invokes once per item with the item's context. Unlike the built-in types it may appear several times, each under a distinct `name` — outcomes and aggregates key on the name. `decider` grades each item with a decision of a project decider: its `score` expression reads the decision's `answers`. Like `tool`, it keys on its `name` and may appear several times.
    - union
      - ExactMatchScorer
        - `type` 'exact_match', required
      - ContainsScorer
        - `type` 'contains', required
        - `value` string, required
        - `case_sensitive` boolean
      - JsonLogicScorer
        - `type` 'json_logic', required
        - `expression` object, required: A JSON Logic expression; a truthy result scores 1
      - OutputSchemaScorer
        - `type` 'output_schema', required
        - `schema` object: JSON Schema the structured output is validated against. Frozen here so two runs stay comparable; falls back to the agent's `output_schema` when omitted.
      - EmbeddingSimilarityScorer
        - `type` 'embedding_similarity', required
        - `pass_threshold` number, required: The item passes this scorer when the cosine similarity between the embeddings of the output text and `expected_output` is greater than or equal to this value. Required.
      - LlmJudgeScorer
        - `type` 'llm_judge', required
        - `prompt` string, required: The judge prompt. `{{input}}`, `{{output}}` and `{{expected}}` are replaced with the item's input messages, the agent's output text, and the item's `expected_output`. Slots are filled in one pass, so a slot value that itself contains `{{output}}` is not re-expanded. The judge must answer with a JSON object carrying a numeric `score` between 0 and 1 and an optional `reasoning` string; a reply that does not marks the **item** errored, never the run failed and never a score of 0.
        - `pass_threshold` number, required: The item passes this scorer when the judge's score is greater than or equal to this value. Required.
        - `ai_provider_id` string, nullable: The AI provider that runs the judge; it must belong to the eval's project. Omit to use the project's default model route.
        - `model` string, nullable: Overrides the provider's default model. Pinned per scorer, because deltas between runs judged by different models are not comparable.
      - ToolScorer: A custom scoring algorithm — a server-callable project tool the engine invokes once per item. The tool receives the same variables a `json_logic` expression reads — `input`, `output`, `object` (when the agent has an `output_schema`), `expected`, and `item.metadata` — with `preset_parameters` merged in at the top level, and must answer with a JSON object carrying a numeric `score` between 0 and 1, an optional boolean `passed`, and an optional `reasoning` string. A malformed answer or a failed call marks the **item** errored, never the run failed and never a score of 0.
        - `type` 'tool', required
        - `name` string, required: Keys this scorer's outcomes and aggregate scores, so it must be unique within the eval and must not shadow a built-in scorer type. Unlike the built-in types, several `tool` scorers may coexist under distinct names.
        - `tool_id` string, required: The tool that scores each item. It must belong to the eval's project and be server-callable (`http`, `mcp`, or `pipeline` — a `client` tool pauses for a calling client an eval run does not have).
        - `action` string, nullable: The operation to invoke; required when the tool type is `mcp`.
        - `preset_parameters` object, nullable: Fixed values merged into every call's input at the top level. The engine-injected keys (`input`, `output`, `object`, `expected`, `item`) are reserved and rejected.
        - `pass_threshold` number, nullable: Fallback verdict cutoff when the tool answers without a `passed` flag: the item passes this scorer when `score` is greater than or equal to this value. A tool-returned `passed` always wins. When the tool omits `passed` and no threshold is set, the item is recorded as errored — the scorer produced no verdict.
      - DeciderScorer: Grades each item with a decision of a project decider, under the decider version pinned when the run started (`decider_versions` on the run). The decision is requested as the run, with `metadata` naming the eval, run and dataset item. A failed decision, or a `score` that is not a number between 0 and 1, marks the **item** errored, never a score of 0.
        - `type` 'decider', required
        - `name` string, required: Keys this scorer's outcomes and aggregate scores, so it must be unique within the eval and must not shadow a built-in scorer type.
        - `decider_id` string, required: The decider that judges each item; it must belong to the eval's project.
        - `input` unknown
        - `score` unknown, required
        - `pass_threshold` number, required: The item passes this scorer when `score` is greater than or equal to this value. Required: a score read from a distribution says nothing about where "good enough" is.
  - `pass_threshold` number, nullable: 0–1. The run passes iff its pass rate — passed items over non-errored items — is at least this. Null reports scores without gating on them.
  - `group_by` string, nullable: A key of the items' `metadata`. A run rolls its scores up per string value of that key in `aggregate_scores.grouping`. Null reports no grouping.

## Response `201`

Eval created successfully

- Eval
  - `id` string
  - `project_id` string
  - `name` string
  - `agent_id` string
  - `dataset_id` string
  - `scorers` union[]: Scorer configs, a discriminated union on `type`. Each type may appear at most once. Every scorer produces `{ score: 0–1, passed: boolean }`; binary scorers emit 0 or 1. `exact_match` compares the trimmed output text to `expected_output`. `contains` looks for `value` in the output text. `json_logic` evaluates `expression` over `{ input, output, object, expected, item.metadata }`, where `object` is the structured output (absent when the agent has no `output_schema`). `output_schema` validates the structured output against the scorer's own `schema`, falling back to the agent's; it requires the agent to carry an `output_schema`, because without one the platform emits no structured output and every item would score 0. `llm_judge` grades the output with a model completion, returning a continuous score plus its `reasoning`. Its `pass_threshold` is required: a continuous score says nothing about where "good enough" is, and a defaulted cutoff would silently decide the gate. `embedding_similarity` embeds the output text and `expected_output` with the platform's configured embedding model (`EMBEDDING_PROVIDER` / `EMBEDDING_MODEL` — the same stack document ingestion uses) and scores their cosine similarity, clamped to 0-1. Its `pass_threshold` is required for the same reason as the judge's. An item without an `expected_output` scores 0; an embedding backend failure marks the **item** errored, never a score of 0. `tool` runs a custom scoring algorithm: a server-callable project tool the engine invokes once per item with the item's context. Unlike the built-in types it may appear several times, each under a distinct `name` — outcomes and aggregates key on the name. `decider` grades each item with a decision of a project decider: its `score` expression reads the decision's `answers`. Like `tool`, it keys on its `name` and may appear several times.
    - union
      - ExactMatchScorer
        - `type` 'exact_match', required
      - ContainsScorer
        - `type` 'contains', required
        - `value` string, required
        - `case_sensitive` boolean
      - JsonLogicScorer
        - `type` 'json_logic', required
        - `expression` object, required: A JSON Logic expression; a truthy result scores 1
      - OutputSchemaScorer
        - `type` 'output_schema', required
        - `schema` object: JSON Schema the structured output is validated against. Frozen here so two runs stay comparable; falls back to the agent's `output_schema` when omitted.
      - EmbeddingSimilarityScorer
        - `type` 'embedding_similarity', required
        - `pass_threshold` number, required: The item passes this scorer when the cosine similarity between the embeddings of the output text and `expected_output` is greater than or equal to this value. Required.
      - LlmJudgeScorer
        - `type` 'llm_judge', required
        - `prompt` string, required: The judge prompt. `{{input}}`, `{{output}}` and `{{expected}}` are replaced with the item's input messages, the agent's output text, and the item's `expected_output`. Slots are filled in one pass, so a slot value that itself contains `{{output}}` is not re-expanded. The judge must answer with a JSON object carrying a numeric `score` between 0 and 1 and an optional `reasoning` string; a reply that does not marks the **item** errored, never the run failed and never a score of 0.
        - `pass_threshold` number, required: The item passes this scorer when the judge's score is greater than or equal to this value. Required.
        - `ai_provider_id` string, nullable: The AI provider that runs the judge; it must belong to the eval's project. Omit to use the project's default model route.
        - `model` string, nullable: Overrides the provider's default model. Pinned per scorer, because deltas between runs judged by different models are not comparable.
      - ToolScorer: A custom scoring algorithm — a server-callable project tool the engine invokes once per item. The tool receives the same variables a `json_logic` expression reads — `input`, `output`, `object` (when the agent has an `output_schema`), `expected`, and `item.metadata` — with `preset_parameters` merged in at the top level, and must answer with a JSON object carrying a numeric `score` between 0 and 1, an optional boolean `passed`, and an optional `reasoning` string. A malformed answer or a failed call marks the **item** errored, never the run failed and never a score of 0.
        - `type` 'tool', required
        - `name` string, required: Keys this scorer's outcomes and aggregate scores, so it must be unique within the eval and must not shadow a built-in scorer type. Unlike the built-in types, several `tool` scorers may coexist under distinct names.
        - `tool_id` string, required: The tool that scores each item. It must belong to the eval's project and be server-callable (`http`, `mcp`, or `pipeline` — a `client` tool pauses for a calling client an eval run does not have).
        - `action` string, nullable: The operation to invoke; required when the tool type is `mcp`.
        - `preset_parameters` object, nullable: Fixed values merged into every call's input at the top level. The engine-injected keys (`input`, `output`, `object`, `expected`, `item`) are reserved and rejected.
        - `pass_threshold` number, nullable: Fallback verdict cutoff when the tool answers without a `passed` flag: the item passes this scorer when `score` is greater than or equal to this value. A tool-returned `passed` always wins. When the tool omits `passed` and no threshold is set, the item is recorded as errored — the scorer produced no verdict.
      - DeciderScorer: Grades each item with a decision of a project decider, under the decider version pinned when the run started (`decider_versions` on the run). The decision is requested as the run, with `metadata` naming the eval, run and dataset item. A failed decision, or a `score` that is not a number between 0 and 1, marks the **item** errored, never a score of 0.
        - `type` 'decider', required
        - `name` string, required: Keys this scorer's outcomes and aggregate scores, so it must be unique within the eval and must not shadow a built-in scorer type.
        - `decider_id` string, required: The decider that judges each item; it must belong to the eval's project.
        - `input` unknown
        - `score` unknown, required
        - `pass_threshold` number, required: The item passes this scorer when `score` is greater than or equal to this value. Required: a score read from a distribution says nothing about where "good enough" is.
  - `pass_threshold` number, nullable
  - `group_by` string, nullable: The item `metadata` key a run rolls its scores up by; null when the eval groups nothing
  - `created_at` string, date-time
  - `updated_at` string, date-time

## Other responses

- `400`: Bad request (unknown scorer type, cross-project reference, invalid threshold)
- `401`: Unauthorized
- `403`: Forbidden
- `409`: An eval with that name already exists in the project
- `500`: Internal server error

---

[API](https://skmtc.dev/naturali/apis/naturali-ai-api.md) · [All operations](https://skmtc.dev/naturali/apis/naturali-ai-api/llms.txt) · [OpenAPI document](https://skmtc.dev/naturali/apis/naturali-ai-api/revisions/e4905b78f422?raw)