---
title: "List evals"
method: GET
path: "/v1/projects/{project_id}/evals"
tags: ["Evaluations"]
---

# List evals

`GET /v1/projects/{project_id}/evals`

## Request

- Base URL: `https://api.naturali.ai`
- URL: `https://api.naturali.ai/v1/projects/{project_id}/evals`
- Auth: one of:
  - HTTP bearer
  - OAuth 2 (authorization code, authorization URL `https://api.naturali.ai/authorize`, token URL `https://api.naturali.ai/token`, scopes: `mcp:access`)

Returns the evals defined in a project

## Query parameters

- `limit` integer
- `offset` integer

## Response `200`

List of evals

- object
  - `data` Eval[], required
    - `id` string
    - `project_id` string
    - `name` string
    - `agent_id` string
    - `dataset_id` string
    - `scorers` union[]: Scorer configs, a discriminated union on `type`. Each type may appear at most once. Every scorer produces `{ score: 0–1, passed: boolean }`; binary scorers emit 0 or 1. `exact_match` compares the trimmed output text to `expected_output`. `contains` looks for `value` in the output text. `json_logic` evaluates `expression` over `{ input, output, object, expected, item.metadata }`, where `object` is the structured output (absent when the agent has no `output_schema`). `output_schema` validates the structured output against the scorer's own `schema`, falling back to the agent's; it requires the agent to carry an `output_schema`, because without one the platform emits no structured output and every item would score 0. `llm_judge` grades the output with a model completion, returning a continuous score plus its `reasoning`. Its `pass_threshold` is required: a continuous score says nothing about where "good enough" is, and a defaulted cutoff would silently decide the gate. `embedding_similarity` embeds the output text and `expected_output` with the platform's configured embedding model (`EMBEDDING_PROVIDER` / `EMBEDDING_MODEL` — the same stack document ingestion uses) and scores their cosine similarity, clamped to 0-1. Its `pass_threshold` is required for the same reason as the judge's. An item without an `expected_output` scores 0; an embedding backend failure marks the **item** errored, never a score of 0. `tool` runs a custom scoring algorithm: a server-callable project tool the engine invokes once per item with the item's context. Unlike the built-in types it may appear several times, each under a distinct `name` — outcomes and aggregates key on the name. `decider` grades each item with a decision of a project decider: its `score` expression reads the decision's `answers`. Like `tool`, it keys on its `name` and may appear several times.
      - union
        - ExactMatchScorer
          - `type` 'exact_match', required
        - ContainsScorer
          - `type` 'contains', required
          - `value` string, required
          - `case_sensitive` boolean
        - JsonLogicScorer
          - `type` 'json_logic', required
          - `expression` object, required: A JSON Logic expression; a truthy result scores 1
        - OutputSchemaScorer
          - `type` 'output_schema', required
          - `schema` object: JSON Schema the structured output is validated against. Frozen here so two runs stay comparable; falls back to the agent's `output_schema` when omitted.
        - EmbeddingSimilarityScorer
          - `type` 'embedding_similarity', required
          - `pass_threshold` number, required: The item passes this scorer when the cosine similarity between the embeddings of the output text and `expected_output` is greater than or equal to this value. Required.
        - LlmJudgeScorer
          - `type` 'llm_judge', required
          - `prompt` string, required: The judge prompt. `{{input}}`, `{{output}}` and `{{expected}}` are replaced with the item's input messages, the agent's output text, and the item's `expected_output`. Slots are filled in one pass, so a slot value that itself contains `{{output}}` is not re-expanded. The judge must answer with a JSON object carrying a numeric `score` between 0 and 1 and an optional `reasoning` string; a reply that does not marks the **item** errored, never the run failed and never a score of 0.
          - `pass_threshold` number, required: The item passes this scorer when the judge's score is greater than or equal to this value. Required.
          - `ai_provider_id` string, nullable: The AI provider that runs the judge; it must belong to the eval's project. Omit to use the project's default model route.
          - `model` string, nullable: Overrides the provider's default model. Pinned per scorer, because deltas between runs judged by different models are not comparable.
        - ToolScorer: A custom scoring algorithm — a server-callable project tool the engine invokes once per item. The tool receives the same variables a `json_logic` expression reads — `input`, `output`, `object` (when the agent has an `output_schema`), `expected`, and `item.metadata` — with `preset_parameters` merged in at the top level, and must answer with a JSON object carrying a numeric `score` between 0 and 1, an optional boolean `passed`, and an optional `reasoning` string. A malformed answer or a failed call marks the **item** errored, never the run failed and never a score of 0.
          - `type` 'tool', required
          - `name` string, required: Keys this scorer's outcomes and aggregate scores, so it must be unique within the eval and must not shadow a built-in scorer type. Unlike the built-in types, several `tool` scorers may coexist under distinct names.
          - `tool_id` string, required: The tool that scores each item. It must belong to the eval's project and be server-callable (`http`, `mcp`, or `pipeline` — a `client` tool pauses for a calling client an eval run does not have).
          - `action` string, nullable: The operation to invoke; required when the tool type is `mcp`.
          - `preset_parameters` object, nullable: Fixed values merged into every call's input at the top level. The engine-injected keys (`input`, `output`, `object`, `expected`, `item`) are reserved and rejected.
          - `pass_threshold` number, nullable: Fallback verdict cutoff when the tool answers without a `passed` flag: the item passes this scorer when `score` is greater than or equal to this value. A tool-returned `passed` always wins. When the tool omits `passed` and no threshold is set, the item is recorded as errored — the scorer produced no verdict.
        - DeciderScorer: Grades each item with a decision of a project decider, under the decider version pinned when the run started (`decider_versions` on the run). The decision is requested as the run, with `metadata` naming the eval, run and dataset item. A failed decision, or a `score` that is not a number between 0 and 1, marks the **item** errored, never a score of 0.
          - `type` 'decider', required
          - `name` string, required: Keys this scorer's outcomes and aggregate scores, so it must be unique within the eval and must not shadow a built-in scorer type.
          - `decider_id` string, required: The decider that judges each item; it must belong to the eval's project.
          - `input` unknown
          - `score` unknown, required
          - `pass_threshold` number, required: The item passes this scorer when `score` is greater than or equal to this value. Required: a score read from a distribution says nothing about where "good enough" is.
    - `pass_threshold` number, nullable
    - `group_by` string, nullable: The item `metadata` key a run rolls its scores up by; null when the eval groups nothing
    - `created_at` string, date-time
    - `updated_at` string, date-time
  - `total` integer, required
  - `limit` integer, required
  - `offset` integer, required

## Other responses

- `401`: Unauthorized
- `403`: Forbidden
- `500`: Internal server error

---

[API](https://skmtc.dev/naturali/apis/naturali-ai-api.md) · [All operations](https://skmtc.dev/naturali/apis/naturali-ai-api/llms.txt) · [OpenAPI document](https://skmtc.dev/naturali/apis/naturali-ai-api/revisions/e4905b78f422?raw)