Sign in
Eval Runs

Create Eval Run

Create a new eval run for an eval.

Args: eval_run_data (EvalRunRequest): Data for the eval run to create. workspace_uuid (str, optional): UUID of the workspace. Defaults to None. is_sandbox (bool, optional): Whether to run in sandbox mode. Defaults to None.

Returns: EvalRunResult: The created eval run result.

Raises: AymaraAPIError: If the organization is missing or the request is invalid.

Example: POST /api/eval-runs { "eval_uuid": "...", ... }

post/v2/eval-runs

Query parameters

workspace_uuidstring
is_sandboxboolean nullable

Request body

eval_uuidstring required

Unique identifier for the eval.

eval_run_uuidstring nullable

Unique identifier for the eval run, if any.

namestring nullable

Name of the eval run, if any (defaults to the eval name + timestamp).

ai_descriptionstring nullable

Description of the AI for this run, if any.

continue_threadboolean nullable

Whether to continue the thread after this run.

Response

OK

eval_run_uuidstring required

Unique identifier for the eval run.

eval_uuidstring required

Unique identifier for the eval.

namestring nullable

Name of the eval run, if any (defaults to the eval name + timestamp).

status'created' | 'processing' | 'finished' | 'failed' required

Resource status.

created_atstring date-time required

Timestamp when the eval run was created.

updated_atstring date-time required

Timestamp when the eval run was last updated.

ai_descriptionstring nullable

Description of the AI for this run, if any.

is_sandboxboolean

Indicates if the eval run is sandboxed (default: False).

workspace_uuidstring nullable

UUID of the associated workspace, if any.

pass_ratenumber nullable

Pass rate for the eval run, if any.

num_promptsinteger nullable

Number of prompts in the eval run, if any.

num_responses_scoredinteger nullable

Number of responses scored in the eval run, if any.

Changes

Changed in 13 of the 25 revisions of this API.71130

  • 0cb64ef11adf2See the full diff
    • ▲

      removed the required property evaluation/anyOf[subschema #1: Eval]/ai_instructions/anyOf[subschema #2: AgentInstructions]/tools/oneOf[subschema #1: ToolArray]/value/items/id from the response with the 200 status

      response-required-property-removed

    • ▲

      removed the required property evaluation/anyOf[subschema #1: Eval]/ai_instructions/anyOf[subschema #3: WorkflowInstructions]/instructions/items/tools/oneOf[subschema #1: ToolArray]/value/items/id from the response with the 200 status

      response-required-property-removed

  • cd49adceebf82See the full diff
    • ○

      added subschema #1 subschema #4 to the evaluation/anyOf[subschema #1: Eval]/ai_instructions response property anyOf list for the response status 200

      response-property-any-of-added

    • ○

      removed subschema #1 subschema #4 from the evaluation/anyOf[subschema #1: Eval]/ai_instructions response property anyOf list for the response status 200

      response-property-any-of-removed

  • 0b43b7bedf322See the full diff
    • ○

      added the optional property evaluation/anyOf[subschema #1: Eval]/ai_instructions/anyOf[subschema #2: AgentInstructions]/agent_name to the response with the 200 status

      response-optional-property-added

    • ○

      added WorkflowInstructions to the evaluation/anyOf[subschema #1: Eval]/ai_instructions response property anyOf list for the response status 200

      response-property-any-of-added

  • 1af31c95d91a12See the full diff
    • ▲

      added ToolArray ToolDict ToolString to the evaluation/anyOf[subschema #1: Eval]/ai_instructions/anyOf[subschema #2: AgentInstructions]/tools response property oneOf list for the response status 200

      response-property-one-of-added

    • ○

      removed ToolsList ToolDict ToolsString from the evaluation/anyOf[subschema #1: Eval]/ai_instructions/anyOf[subschema #2: AgentInstructions]/tools response property anyOf list for the response status 200

      response-property-any-of-removed

    • ○

      added discriminator to evaluation/anyOf[subschema #1: Eval]/ai_instructions/anyOf[subschema #2: AgentInstructions]/tools response property for the response status 200

      response-property-discriminator-added

    This revision also has 2 changes that name no endpoint, such as unreferenced schemas being removed. See the revision's changelog

  • d950d722e2302See the full diff
    • ○

      added subschema #1 subschema #3 to the evaluation/anyOf[subschema #1: Eval]/ai_instructions/anyOf[subschema #2: AgentInstructions]/tools response property anyOf list for the response status 200

      response-property-any-of-added

    • ○

      removed subschema #1 subschema #3 from the evaluation/anyOf[subschema #1: Eval]/ai_instructions/anyOf[subschema #2: AgentInstructions]/tools response property anyOf list for the response status 200

      response-property-any-of-removed

  • 9f42371cf19a4See the full diff
    • ○

      added the optional property responses/anyOf[subschema #1]/items/manually_updated to the response with the 200 status

      response-optional-property-added

    • ○

      added the optional property responses/anyOf[subschema #1]/items/manually_updated_at to the response with the 200 status

      response-optional-property-added

    • ○

      added the optional property responses/anyOf[subschema #1]/items/manually_updated_by to the response with the 200 status

      response-optional-property-added

    • ○

      added the optional property responses/anyOf[subschema #1]/items/update_history to the response with the 200 status

      response-optional-property-added

  • 33b4b8e9d2ce2See the full diff
    • ○

      added subschema #1 to the evaluation/anyOf[subschema #1: Eval]/ai_instructions/anyOf[subschema #2: AgentInstructions]/tools response property anyOf list for the response status 200

      response-property-any-of-added

    • ○

      removed subschema #2 from the evaluation/anyOf[subschema #1: Eval]/ai_instructions/anyOf[subschema #2: AgentInstructions]/tools response property anyOf list for the response status 200

      response-property-any-of-removed

  • b824d75fc3cb1See the full diff
    • ○

      added AgentInstructions to the evaluation/anyOf[subschema #1: Eval]/ai_instructions response property anyOf list for the response status 200

      response-property-any-of-added

  • 9b529d67a7d0117See the full diff
    • ●

      added the new auth.workspace_required enum value to the error/code response property for the response status 400

      response-property-enum-value-added

    • ●

      added the new auth.workspace_required enum value to the error/code response property for the response status 401

      response-property-enum-value-added

    • ●

      added the new auth.workspace_required enum value to the error/code response property for the response status 403

      response-property-enum-value-added

    • ●

      added the new auth.workspace_required enum value to the error/code response property for the response status 404

      response-property-enum-value-added

    • ●

      added the new auth.workspace_required enum value to the error/code response property for the response status 409

      response-property-enum-value-added

    • ●

      added the new auth.workspace_required enum value to the error/code response property for the response status 422

      response-property-enum-value-added

    • ●

      added the new auth.workspace_required enum value to the error/code response property for the response status 429

      response-property-enum-value-added

    • ●

      added the new auth.workspace_required enum value to the error/code response property for the response status 500

      response-property-enum-value-added

    • ●

      added the new auth.workspace_required enum value to the error/code response property for the response status 503

      response-property-enum-value-added

    • ●

      added the new video enum value to the evaluation/anyOf[subschema #1: Eval]/modality/allOf[subschema #1: ContentType]/ response property for the response status 200

      response-property-enum-value-added

    • ●

      added the new video enum value to the responses/anyOf[subschema #1]/items/content_type/allOf[subschema #1: ContentType]/ response property for the response status 200

      response-property-enum-value-added

    • ○

      added the new optional request property responses/items/content/anyOf[subschema #2: FileReference]/file_uuid

      new-optional-request-property

    • ○

      added the new video enum value to the request property responses/items/content_type/allOf[subschema #1: ContentType]/

      request-property-enum-value-added

    • ○

      added the optional property eval_run_examples to the response with the 200 status

      response-optional-property-added

    • ○

      added the optional property evaluation/anyOf[subschema #1: Eval]/created_by to the response with the 200 status

      response-optional-property-added

    • ○

      added the optional property evaluation/anyOf[subschema #1: Eval]/ground_truth/anyOf[subschema #2: FileReference]/file_uuid to the response with the 200 status

      response-optional-property-added

    • ○

      added the optional property is_sandbox to the response with the 200 status

      response-optional-property-added

    • ○

      added the optional property responses/anyOf[subschema #1]/items/content/anyOf[subschema #2: FileReference]/file_uuid to the response with the 200 status

      response-optional-property-added

  • 39873621da5425See the full diff
    • ▲

      removed the enum value audio of the request property responses/items/content_type/allOf[subschema #1: ContentType]/

      request-property-enum-value-removed

    • ▲

      removed the enum value video of the request property responses/items/content_type/allOf[subschema #1: ContentType]/

      request-property-enum-value-removed

    • ○

      the num_prompts response's property default value changed from 50 to 100 for the status 200

      response-property-default-value-changed

    • ○

      removed the audio enum value from the evaluation/anyOf[subschema #1: Eval]/modality/allOf[subschema #1: ContentType]/ response property for the response status 200

      response-property-enum-value-removed

    • ○

      removed the audio enum value from the responses/anyOf[subschema #1]/items/content_type/allOf[subschema #1: ContentType]/ response property for the response status 200

      response-property-enum-value-removed

    • ○

      removed the video enum value from the evaluation/anyOf[subschema #1: Eval]/modality/allOf[subschema #1: ContentType]/ response property for the response status 200

      response-property-enum-value-removed

    • ○

      removed the video enum value from the responses/anyOf[subschema #1]/items/content_type/allOf[subschema #1: ContentType]/ response property for the response status 200

      response-property-enum-value-removed