---
title: "Get an eval"
method: GET
path: "/v1/projects/{project_id}/evals/{eval_id}"
tags: ["Evaluations"]
---

# Get an eval

`GET /v1/projects/{project_id}/evals/{eval_id}`

## Request

- Base URL: `https://api.naturali.ai`
- URL: `https://api.naturali.ai/v1/projects/{project_id}/evals/{eval_id}`
- Auth: one of:
  - HTTP bearer
  - OAuth 2 (authorization code, authorization URL `https://api.naturali.ai/authorize`, token URL `https://api.naturali.ai/token`, scopes: `mcp:access`)

Returns a specific eval

## Path parameters

- `eval_id` string, required

## Response `200`

Eval details

- Eval
  - `id` string
  - `project_id` string
  - `name` string
  - `agent_id` string
  - `dataset_id` string
  - `scorers` union[]: Scorer configs, a discriminated union on `type`. Each type may appear at most once. Every scorer produces `{ score: 0–1, passed: boolean }`; binary scorers emit 0 or 1. `exact_match` compares the trimmed output text to `expected_output`. `contains` looks for `value` in the output text. `json_logic` evaluates `expression` over `{ input, output, object, expected, item.metadata }`, where `object` is the structured output (absent when the agent has no `output_schema`). `output_schema` validates the structured output against the scorer's own `schema`, falling back to the agent's; it requires the agent to carry an `output_schema`, because without one the platform emits no structured output and every item would score 0. `llm_judge` grades the output with a model completion, returning a continuous score plus its `reasoning`. Its `pass_threshold` is required: a continuous score says nothing about where "good enough" is, and a defaulted cutoff would silently decide the gate. `embedding_similarity` embeds the output text and `expected_output` with the platform's configured embedding model (`EMBEDDING_PROVIDER` / `EMBEDDING_MODEL` — the same stack document ingestion uses) and scores their cosine similarity, clamped to 0-1. Its `pass_threshold` is required for the same reason as the judge's. An item without an `expected_output` scores 0; an embedding backend failure marks the **item** errored, never a score of 0. `tool` runs a custom scoring algorithm: a server-callable project tool the engine invokes once per item with the item's context. Unlike the built-in types it may appear several times, each under a distinct `name` — outcomes and aggregates key on the name. `decider` grades each item with a decision of a project decider: its `score` expression reads the decision's `answers`. Like `tool`, it keys on its `name` and may appear several times.
    - union
      - ExactMatchScorer
        - `type` 'exact_match', required
      - ContainsScorer
        - `type` 'contains', required
        - `value` string, required
        - `case_sensitive` boolean
      - JsonLogicScorer
        - `type` 'json_logic', required
        - `expression` object, required: A JSON Logic expression; a truthy result scores 1
      - OutputSchemaScorer
        - `type` 'output_schema', required
        - `schema` object: JSON Schema the structured output is validated against. Frozen here so two runs stay comparable; falls back to the agent's `output_schema` when omitted.
      - EmbeddingSimilarityScorer
        - `type` 'embedding_similarity', required
        - `pass_threshold` number, required: The item passes this scorer when the cosine similarity between the embeddings of the output text and `expected_output` is greater than or equal to this value. Required.
      - LlmJudgeScorer
        - `type` 'llm_judge', required
        - `prompt` string, required: The judge prompt. `{{input}}`, `{{output}}` and `{{expected}}` are replaced with the item's input messages, the agent's output text, and the item's `expected_output`. Slots are filled in one pass, so a slot value that itself contains `{{output}}` is not re-expanded. The judge must answer with a JSON object carrying a numeric `score` between 0 and 1 and an optional `reasoning` string; a reply that does not marks the **item** errored, never the run failed and never a score of 0.
        - `pass_threshold` number, required: The item passes this scorer when the judge's score is greater than or equal to this value. Required.
        - `ai_provider_id` string, nullable: The AI provider that runs the judge; it must belong to the eval's project. Omit to use the project's default model route.
        - `model` string, nullable: Overrides the provider's default model. Pinned per scorer, because deltas between runs judged by different models are not comparable.
      - ToolScorer: A custom scoring algorithm — a server-callable project tool the engine invokes once per item. The tool receives the same variables a `json_logic` expression reads — `input`, `output`, `object` (when the agent has an `output_schema`), `expected`, and `item.metadata` — with `preset_parameters` merged in at the top level, and must answer with a JSON object carrying a numeric `score` between 0 and 1, an optional boolean `passed`, and an optional `reasoning` string. A malformed answer or a failed call marks the **item** errored, never the run failed and never a score of 0.
        - `type` 'tool', required
        - `name` string, required: Keys this scorer's outcomes and aggregate scores, so it must be unique within the eval and must not shadow a built-in scorer type. Unlike the built-in types, several `tool` scorers may coexist under distinct names.
        - `tool_id` string, required: The tool that scores each item. It must belong to the eval's project and be server-callable (`http`, `mcp`, or `pipeline` — a `client` tool pauses for a calling client an eval run does not have).
        - `action` string, nullable: The operation to invoke; required when the tool type is `mcp`.
        - `preset_parameters` object, nullable: Fixed values merged into every call's input at the top level. The engine-injected keys (`input`, `output`, `object`, `expected`, `item`) are reserved and rejected.
        - `pass_threshold` number, nullable: Fallback verdict cutoff when the tool answers without a `passed` flag: the item passes this scorer when `score` is greater than or equal to this value. A tool-returned `passed` always wins. When the tool omits `passed` and no threshold is set, the item is recorded as errored — the scorer produced no verdict.
      - DeciderScorer: Grades each item with a decision of a project decider, under the decider version pinned when the run started (`decider_versions` on the run). The decision is requested as the run, with `metadata` naming the eval, run and dataset item. A failed decision, or a `score` that is not a number between 0 and 1, marks the **item** errored, never a score of 0.
        - `type` 'decider', required
        - `name` string, required: Keys this scorer's outcomes and aggregate scores, so it must be unique within the eval and must not shadow a built-in scorer type.
        - `decider_id` string, required: The decider that judges each item; it must belong to the eval's project.
        - `input` unknown
        - `score` unknown, required
        - `pass_threshold` number, required: The item passes this scorer when `score` is greater than or equal to this value. Required: a score read from a distribution says nothing about where "good enough" is.
  - `pass_threshold` number, nullable
  - `group_by` string, nullable: The item `metadata` key a run rolls its scores up by; null when the eval groups nothing
  - `created_at` string, date-time
  - `updated_at` string, date-time

## Other responses

- `401`: Unauthorized
- `403`: Forbidden
- `404`: Eval not found

---

[API](https://skmtc.dev/naturali/apis/naturali-ai-api.md) · [All operations](https://skmtc.dev/naturali/apis/naturali-ai-api/llms.txt) · [OpenAPI document](https://skmtc.dev/naturali/apis/naturali-ai-api/revisions/e4905b78f422?raw)