Sign in
Evals

Create Eval

Create a new eval using an eval type configuration.

Args: eval_request (Eval): Data for the eval to create, including eval type and configuration.

Returns: Eval: The created eval object.

Raises: AymaraAPIError: If the request is invalid.

Example: POST /api/evals { "eval_type": "...", "workspace_uuid": "...", ... }

post/v2/evals

Request body

eval_uuidstring nullable

Unique identifier for the evaluation.

namestring nullable

Name of the evaluation.

created_bystring nullable

Name of the user who created the evaluation.

ai_descriptionstring required

Description of the AI under evaluation.

eval_typestring required

Type of the eval (safety, accuracy, etc.)

eval_instructionsstring nullable

Additional instructions for the eval, if any.

languagestring nullable

Language code for the eval (default: "en").

modality'text' | 'image' | 'video'

Content type for AI interactions.

num_promptsinteger nullable

Number of prompts/questions in the eval (default: 50).

is_jailbreakboolean

Indicates if the eval is a jailbreak test.

is_sandboxboolean

Indicates if the eval results are sandboxed.

workspace_uuidstring nullable

UUID of the associated workspace, if any.

status'created' | 'processing' | 'finished' | 'failed'

Resource status.

created_atstring date-time nullable

Timestamp when the eval was created.

updated_atstring date-time nullable

Timestamp when the eval was last updated.

Response

Created

eval_uuidstring nullable

Unique identifier for the evaluation.

namestring nullable

Name of the evaluation.

created_bystring nullable

Name of the user who created the evaluation.

ai_descriptionstring required

Description of the AI under evaluation.

eval_typestring required

Type of the eval (safety, accuracy, etc.)

eval_instructionsstring nullable

Additional instructions for the eval, if any.

languagestring nullable

Language code for the eval (default: "en").

modality'text' | 'image' | 'video'

Content type for AI interactions.

num_promptsinteger nullable

Number of prompts/questions in the eval (default: 50).

is_jailbreakboolean

Indicates if the eval is a jailbreak test.

is_sandboxboolean

Indicates if the eval results are sandboxed.

workspace_uuidstring nullable

UUID of the associated workspace, if any.

status'created' | 'processing' | 'finished' | 'failed'

Resource status.

created_atstring date-time nullable

Timestamp when the eval was created.

updated_atstring date-time nullable

Timestamp when the eval was last updated.

Changes

Changed in 11 of the 25 revisions of this API.111232

  • 0cb64ef11adf22See the full diff
    • ▲

      removed the required property ai_instructions/anyOf[subschema #2: AgentInstructions]/tools/oneOf[subschema #1: ToolArray]/value/items/id from the response with the 201 status

      response-required-property-removed

    • ▲

      removed the required property ai_instructions/anyOf[subschema #3: WorkflowInstructions]/instructions/items/tools/oneOf[subschema #1: ToolArray]/value/items/id from the response with the 201 status

      response-required-property-removed

    • ●

      removed the request property ai_instructions/anyOf[subschema #2: AgentInstructions]/tools/oneOf[subschema #1: ToolArray]/value/items/id

      request-property-removed

    • ●

      removed the request property ai_instructions/anyOf[subschema #3: WorkflowInstructions]/instructions/items/tools/oneOf[subschema #1: ToolArray]/value/items/id

      request-property-removed

  • cd49adceebf813See the full diff
    • ▲

      removed subschema #1 subschema #4 from the ai_instructions request property anyOf list

      request-property-any-of-removed

    • ○

      added subschema #1 subschema #4 to the ai_instructions request property anyOf list

      request-property-any-of-added

    • ○

      added subschema #1 subschema #4 to the ai_instructions response property anyOf list for the response status 201

      response-property-any-of-added

    • ○

      removed subschema #1 subschema #4 from the ai_instructions response property anyOf list for the response status 201

      response-property-any-of-removed

  • 0b43b7bedf324See the full diff
    • ○

      added the new optional request property ai_instructions/anyOf[subschema #2: AgentInstructions]/agent_name

      new-optional-request-property

    • ○

      added WorkflowInstructions to the ai_instructions request property anyOf list

      request-property-any-of-added

    • ○

      added the optional property ai_instructions/anyOf[subschema #2: AgentInstructions]/agent_name to the response with the 201 status

      response-optional-property-added

    • ○

      added WorkflowInstructions to the ai_instructions response property anyOf list for the response status 201

      response-property-any-of-added

  • 1af31c95d91a24See the full diff
    • ▲

      removed ToolsList ToolDict ToolsString from the ai_instructions/anyOf[subschema #2: AgentInstructions]/tools request property anyOf list

      request-property-any-of-removed

    • ▲

      added ToolArray ToolDict ToolString to the ai_instructions/anyOf[subschema #2: AgentInstructions]/tools response property oneOf list for the response status 201

      response-property-one-of-added

    • ○

      added discriminator to ai_instructions/anyOf[subschema #2: AgentInstructions]/tools request property

      request-property-discriminator-added

    • ○

      added ToolArray ToolDict ToolString to the ai_instructions/anyOf[subschema #2: AgentInstructions]/tools request property oneOf list

      request-property-one-of-added

    • ○

      removed ToolsList ToolDict ToolsString from the ai_instructions/anyOf[subschema #2: AgentInstructions]/tools response property anyOf list for the response status 201

      response-property-any-of-removed

    • ○

      added discriminator to ai_instructions/anyOf[subschema #2: AgentInstructions]/tools response property for the response status 201

      response-property-discriminator-added

    This revision also has 2 changes that name no endpoint, such as unreferenced schemas being removed. See the revision's changelog

  • d950d722e23013See the full diff
    • ▲

      removed subschema #1 subschema #3 from the ai_instructions/anyOf[subschema #2: AgentInstructions]/tools request property anyOf list

      request-property-any-of-removed

    • ○

      added subschema #1 subschema #3 to the ai_instructions/anyOf[subschema #2: AgentInstructions]/tools request property anyOf list

      request-property-any-of-added

    • ○

      added subschema #1 subschema #3 to the ai_instructions/anyOf[subschema #2: AgentInstructions]/tools response property anyOf list for the response status 201

      response-property-any-of-added

    • ○

      removed subschema #1 subschema #3 from the ai_instructions/anyOf[subschema #2: AgentInstructions]/tools response property anyOf list for the response status 201

      response-property-any-of-removed

  • 33b4b8e9d2ce13See the full diff
    • ▲

      removed subschema #2 from the ai_instructions/anyOf[subschema #2: AgentInstructions]/tools request property anyOf list

      request-property-any-of-removed

    • ○

      added subschema #1 to the ai_instructions/anyOf[subschema #2: AgentInstructions]/tools request property anyOf list

      request-property-any-of-added

    • ○

      added subschema #1 to the ai_instructions/anyOf[subschema #2: AgentInstructions]/tools response property anyOf list for the response status 201

      response-property-any-of-added

    • ○

      removed subschema #2 from the ai_instructions/anyOf[subschema #2: AgentInstructions]/tools response property anyOf list for the response status 201

      response-property-any-of-removed

  • b824d75fc3cb2See the full diff
    • ○

      added AgentInstructions to the ai_instructions request property anyOf list

      request-property-any-of-added

    • ○

      added AgentInstructions to the ai_instructions response property anyOf list for the response status 201

      response-property-any-of-added

  • 9b529d67a7d0105See the full diff
    • ●

      added the new auth.workspace_required enum value to the error/code response property for the response status 400

      response-property-enum-value-added

    • ●

      added the new auth.workspace_required enum value to the error/code response property for the response status 401

      response-property-enum-value-added

    • ●

      added the new auth.workspace_required enum value to the error/code response property for the response status 403

      response-property-enum-value-added

    • ●

      added the new auth.workspace_required enum value to the error/code response property for the response status 404

      response-property-enum-value-added

    • ●

      added the new auth.workspace_required enum value to the error/code response property for the response status 409

      response-property-enum-value-added

    • ●

      added the new auth.workspace_required enum value to the error/code response property for the response status 422

      response-property-enum-value-added

    • ●

      added the new auth.workspace_required enum value to the error/code response property for the response status 429

      response-property-enum-value-added

    • ●

      added the new auth.workspace_required enum value to the error/code response property for the response status 500

      response-property-enum-value-added

    • ●

      added the new auth.workspace_required enum value to the error/code response property for the response status 503

      response-property-enum-value-added

    • ●

      added the new video enum value to the modality/allOf[subschema #1: ContentType]/ response property for the response status 201

      response-property-enum-value-added

    • ○

      added the new optional request property created_by

      new-optional-request-property

    • ○

      added the new optional request property ground_truth/anyOf[subschema #2: FileReference]/file_uuid

      new-optional-request-property

    • ○

      added the new video enum value to the request property modality/allOf[subschema #1: ContentType]/

      request-property-enum-value-added

    • ○

      added the optional property created_by to the response with the 201 status

      response-optional-property-added

    • ○

      added the optional property ground_truth/anyOf[subschema #2: FileReference]/file_uuid to the response with the 201 status

      response-optional-property-added

  • 39873621da5424See the full diff
    • ▲

      removed the enum value audio of the request property modality/allOf[subschema #1: ContentType]/

      request-property-enum-value-removed

    • ▲

      removed the enum value video of the request property modality/allOf[subschema #1: ContentType]/

      request-property-enum-value-removed

    • ○

      the num_prompts request property default value changed from 50 to 100

      request-property-default-value-changed

    • ○

      the num_prompts response's property default value changed from 50 to 100 for the status 201

      response-property-default-value-changed

    • ○

      removed the audio enum value from the modality/allOf[subschema #1: ContentType]/ response property for the response status 201

      response-property-enum-value-removed

    • ○

      removed the video enum value from the modality/allOf[subschema #1: ContentType]/ response property for the response status 201

      response-property-enum-value-removed

  • 13afcad941fd22See the full diff
    • ▲

      the response property name became optional for the status 201

      response-property-became-optional

    • ▲

      response property name list-of-types was widened by adding types null to media type application/json of response 201

      response-property-list-of-types-widened

    • ○

      the request property name became optional

      request-property-became-optional

    • ○

      request property name list-of-types was widened by adding types null to media type application/json

      request-property-list-of-types-widened