---
title: "Run Eval"
method: POST
path: "/v2/eval-runs/-/score-responses"
tags: ["Eval Runs"]
---

# Run Eval

`POST /v2/eval-runs/-/score-responses`

## Request

- Base URL: `https://api.aymara.ai`
- URL: `https://api.aymara.ai/v2/eval-runs/-/score-responses`
- Auth: one of:
  - API key in header `x-api-key`
  - HTTP bearer
- TypeScript: <https://skmtc.dev/aymara-ai/apis/aymara-api/docs/v2/eval-runs/-/score-responses/post.ts> (a function with typed inputs; its response is checked against the schema)

Run the eval with the provided responses.

Args:
    eval_run_data (EvalRunRequest): Data for the eval run, including responses.
    workspace_uuid (str, optional): UUID of the workspace. Defaults to None.
    is_sandbox (bool, optional): Whether to run in sandbox mode. Defaults to False.

Returns:
    EvalRunResult: The result of the eval run after scoring the responses.

Raises:
    AymaraAPIError: If the organization is missing or the request is invalid.

Example:
    POST /api/eval-runs/-/score-responses
    {
        "eval_uuid": "...",
        "responses": [...]
    }

## Query parameters

- `workspace_uuid` string
- `is_sandbox` boolean

## Request body

- EvalRunRequest: Schema for creating or continuing an eval run.
  - `eval_uuid` string, required: Unique identifier for the eval.
  - `eval_run_uuid` string, nullable: Unique identifier for the eval run, if any.
  - `name` string, nullable: Name of the eval run, if any (defaults to the eval name + timestamp).
  - `ai_description` string, nullable: Description of the AI for this run, if any.
  - `continue_thread` boolean, nullable: Whether to continue the thread after this run.
  - `eval_run_examples` EvalRunExample[], nullable: Examples to include with the eval run, if any.
    - `example_uuid` string, nullable: Unique identifier for the example, if any.
    - `type` 'pass' | 'fail', required: Type of the example: "pass" or "fail".
    - `prompt` string, required: Prompt text for the example.
    - `response` string, required: Expected response for the example.
    - `explanation` string, nullable: Explanation for the example, if any.
  - `responses` EvalResponse[], required: List of AI responses to eval prompts.
    - `prompt_uuid` string, required: Unique identifier for the prompt.
    - `thread_uuid` string, nullable: Unique identifier for the thread, if any.
    - `turn_number` integer: Turn number in the conversation (default: 1).
    - `continue_thread` boolean: Whether to continue the thread after this response.
    - `content` union: Content of the AI response or a file reference.
      - string
      - FileReference: Reference to a file, either by file_uuid (preferred) or legacy remote_file_path. When file_uuid is provided, the system will look up the File record and use its remote_file_path. The remote_file_path field is maintained for backwards compatibility.
        - `file_uuid` string, nullable
        - `remote_file_path` string, nullable
    - `content_type` 'text' | 'image' | 'video': Content type for AI interactions.
    - `exclude_from_scoring` boolean: Whether to exclude this response from scoring.
    - `ai_refused` boolean: Whether the AI refused to answer the prompt.

## Response `200`

OK

- EvalRunResult: Schema for returning eval run data.
  - `eval_run_uuid` string, required: Unique identifier for the eval run.
  - `eval_uuid` string, required: Unique identifier for the eval.
  - `name` string, nullable: Name of the eval run, if any (defaults to the eval name + timestamp).
  - `status` 'created' | 'processing' | 'finished' | 'failed', required: Resource status.
  - `created_at` string, date-time, required: Timestamp when the eval run was created.
  - `updated_at` string, date-time, required: Timestamp when the eval run was last updated.
  - `evaluation` Eval: Schema for configuring an Eval based on a eval_type.
    - `eval_uuid` string, nullable: Unique identifier for the evaluation.
    - `name` string, nullable: Name of the evaluation.
    - `created_by` string, nullable: Name of the user who created the evaluation.
    - `ai_description` string, required: Description of the AI under evaluation.
    - `ai_instructions` union: Instructions the AI should follow. String for normal evals, AgentInstructions for single-agent evals, WorkflowInstructions for multi-agent workflows.
      - string
      - AgentInstructions
        - `agent_name` string, nullable
        - `system_prompt` string, required
        - `tools` union: Instructions for the agent, can be a string or a list/dict of tools.
          - ToolArray: Container for an array of tools
            - `type` 'array'
            - `value` Tool[], required
              - …
          - ToolDict: Container for a tool dictionary
            - `type` 'dict'
            - `value` object, required
          - ToolString: Container for string-based tool instructions
            - `type` 'string'
            - `value` string, required
      - WorkflowInstructions: Workflow instructions for multi-agent workflows.
        - `instructions` AgentInstructions[], required: List of agent instructions for the workflow. Must contain at least one agent.
          - `agent_name` string, nullable
          - `system_prompt` string, required
          - `tools` union: Instructions for the agent, can be a string or a list/dict of tools.
            - ToolArray: Container for an array of tools
              - …
            - ToolDict: Container for a tool dictionary
              - …
            - ToolString: Container for string-based tool instructions
              - …
    - `eval_type` string, required: Type of the eval (safety, accuracy, etc.)
    - `eval_instructions` string, nullable: Additional instructions for the eval, if any.
    - `language` string, nullable: Language code for the eval (default: "en").
    - `modality` 'text' | 'image' | 'video': Content type for AI interactions.
    - `ground_truth` union: Ground truth data or reference file, if any.
      - string
      - FileReference: Reference to a file, either by file_uuid (preferred) or legacy remote_file_path. When file_uuid is provided, the system will look up the File record and use its remote_file_path. The remote_file_path field is maintained for backwards compatibility.
        - `file_uuid` string, nullable
        - `remote_file_path` string, nullable
    - `num_prompts` integer, nullable: Number of prompts/questions in the eval (default: 50).
    - `prompt_examples` PromptExample[], nullable: List of example prompts for the eval.
      - `content` string, required: Content of the example prompt.
      - `example_uuid` string, nullable: Unique identifier for the example, if any.
      - `type` 'good' | 'bad'
      - `explanation` string, nullable: Explanation for the example, if any.
    - `is_jailbreak` boolean: Indicates if the eval is a jailbreak test.
    - `is_sandbox` boolean: Indicates if the eval results are sandboxed.
    - `workspace_uuid` string, nullable: UUID of the associated workspace, if any.
    - `status` 'created' | 'processing' | 'finished' | 'failed': Resource status.
    - `created_at` string, date-time, nullable: Timestamp when the eval was created.
    - `updated_at` string, date-time, nullable: Timestamp when the eval was last updated.
  - `ai_description` string, nullable: Description of the AI for this run, if any.
  - `is_sandbox` boolean: Indicates if the eval run is sandboxed (default: False).
  - `workspace_uuid` string, nullable: UUID of the associated workspace, if any.
  - `pass_rate` number, nullable: Pass rate for the eval run, if any.
  - `num_prompts` integer, nullable: Number of prompts in the eval run, if any.
  - `num_responses_scored` integer, nullable: Number of responses scored in the eval run, if any.
  - `eval_run_examples` EvalRunExample[], nullable: Examples that were included with the eval run, if any.
    - `example_uuid` string, nullable: Unique identifier for the example, if any.
    - `type` 'pass' | 'fail', required: Type of the example: "pass" or "fail".
    - `prompt` string, required: Prompt text for the example.
    - `response` string, required: Expected response for the example.
    - `explanation` string, nullable: Explanation for the example, if any.
  - `responses` EvalScoredResponse[], nullable: List of scored responses for the eval run, if any.
    - `prompt_uuid` string, required: Unique identifier for the prompt.
    - `thread_uuid` string, nullable: Unique identifier for the thread, if any.
    - `turn_number` integer: Turn number in the conversation (default: 1).
    - `continue_thread` boolean: Whether to continue the thread after this response.
    - `content` union: Content of the AI response or a file reference.
      - string
      - FileReference: Reference to a file, either by file_uuid (preferred) or legacy remote_file_path. When file_uuid is provided, the system will look up the File record and use its remote_file_path. The remote_file_path field is maintained for backwards compatibility.
        - `file_uuid` string, nullable
        - `remote_file_path` string, nullable
    - `content_type` 'text' | 'image' | 'video': Content type for AI interactions.
    - `exclude_from_scoring` boolean: Whether to exclude this response from scoring.
    - `ai_refused` boolean: Whether the AI refused to answer the prompt.
    - `response_uuid` string, nullable: Unique identifier for the response, if any.
    - `explanation` string, nullable: Explanation for the response, if any.
    - `confidence` number, nullable: Confidence score for the response, if any.
    - `is_passed` boolean, nullable: Whether the response passed the evaluation, if any.
    - `next_prompt` EvalPrompt
      - `prompt_uuid` string, required: Unique identifier for the prompt.
      - `thread_uuid` string, nullable: Unique identifier for the thread, if any.
      - `turn_number` integer: Turn number in the conversation (default: 1).
      - `content` string, required: Content of the prompt.
      - `category` string, nullable: Category of the prompt, if any.
    - `manually_updated` boolean: Whether this response was manually updated.
    - `manually_updated_by` UserOutSchema
      - `email` string, required
      - `organization` OrganizationOutSchema
        - `org_uuid` string, required
        - `name` string, required
      - `workspace` WorkspaceOutSchema
        - `workspace_uuid` string, nullable
        - `name` string, nullable
        - `organization_name` string, nullable
        - `organization_uuid` string, nullable
      - `org_admin_emails` string[], nullable
      - `is_admin` boolean, required
      - `is_impersonating` boolean, required
      - `feature_flags` FeatureFlags, required
        - `enable_dashboard` boolean, required
    - `manually_updated_at` string, date-time, nullable: Timestamp when this response was last manually updated.
    - `update_history` AnswerHistoryOut[], nullable: History of manual updates, if requested.
      - `answer_history_uuid` string, required: Unique identifier for the history entry.
      - `updated_by` UserOutSchema
        - `email` string, required
        - `organization` OrganizationOutSchema
          - `org_uuid` string, required
          - `name` string, required
        - `workspace` WorkspaceOutSchema
          - `workspace_uuid` string, nullable
          - `name` string, nullable
          - `organization_name` string, nullable
          - `organization_uuid` string, nullable
        - `org_admin_emails` string[], nullable
        - `is_admin` boolean, required
        - `is_impersonating` boolean, required
        - `feature_flags` FeatureFlags, required
          - `enable_dashboard` boolean, required
      - `previous_is_passed` boolean, nullable: Previous is_passed value.
      - `previous_confidence` number, nullable: Previous confidence value.
      - `previous_explanation` string, nullable: Previous explanation value.
      - `new_is_passed` boolean, nullable: New is_passed value.
      - `new_confidence` number, nullable: New confidence value.
      - `new_explanation` string, nullable: New explanation value.
      - `updated_at` string, date-time, required: Timestamp when the update was made.

## Other responses

- `400`: Bad Request
- `401`: Unauthorized
- `403`: Forbidden
- `404`: Not Found
- `409`: Conflict
- `422`: Unprocessable Entity
- `429`: Too Many Requests
- `500`: Internal Server Error
- `503`: Service Unavailable

## Changes

- **2026-02-08** `0cb64ef11adf`: 2 breaking
  - removed the required property `evaluation/anyOf[subschema #1: Eval]/ai_instructions/anyOf[subschema #2: AgentInstructions]/tools/oneOf[subschema #1: ToolArray]/value/items/id` from the response with the `200` status
  - removed the required property `evaluation/anyOf[subschema #1: Eval]/ai_instructions/anyOf[subschema #3: WorkflowInstructions]/instructions/items/tools/oneOf[subschema #1: ToolArray]/value/items/id` from the response with the `200` status
- **2026-02-08** `cd49adceebf8`: 2 info
  - added `subschema #1, subschema #4` to the `evaluation/anyOf[subschema #1: Eval]/ai_instructions` response property `anyOf` list for the response status `200`
  - removed `subschema #1, subschema #4` from the `evaluation/anyOf[subschema #1: Eval]/ai_instructions` response property `anyOf` list for the response status `200`
- **2026-02-08** `0b43b7bedf32`: 2 info
  - added the optional property `evaluation/anyOf[subschema #1: Eval]/ai_instructions/anyOf[subschema #2: AgentInstructions]/agent_name` to the response with the `200` status
  - added `#/components/schemas/WorkflowInstructions` to the `evaluation/anyOf[subschema #1: Eval]/ai_instructions` response property `anyOf` list for the response status `200`
- …earlier changes not shown

[Operation history](https://skmtc.dev/aymara-ai/apis/aymara-api/changes/v2/eval-runs/-/score-responses/post.md)

---

[API](https://skmtc.dev/aymara-ai/apis/aymara-api.md) - [All operations](https://skmtc.dev/aymara-ai/apis/aymara-api/llms.txt) - [OpenAPI document](https://skmtc.dev/aymara-ai/apis/aymara-api/revisions/0cb64ef11adf?raw)