Sign in
Eval Analysis

Analyze Evals

Analysis for evals with advanced filtering and aggregated statistics.

This endpoint allows analyzing across both eval metadata and score run performance data, providing comprehensive filtering capabilities and aggregated statistics for each eval.

Args: analysis_request (EvalAnalysisRequest): Analysis parameters and filters including: - Eval metadata filters (name, type, status, language, etc.) - Score run performance filters (pass rate, run count, etc.) - Sorting and pagination options

Returns: EvalAnalysisResponse: Paginated results with matching evals and their statistics

Raises: AymaraAPIError: If the request is invalid or analysis parameters are malformed

Example: POST /api/v2/eval_analysis { "name": "safety", "eval_type": "safety", "min_pass_rate": 0.8, "has_score_runs": true, "sort_by": "pass_rate", "sort_order": "desc", "limit": 20, "offset": 0 }

post/v2/eval-analysis

Request body

namestring nullable

Filter by eval names (case-insensitive partial match)

eval_typestring nullable

Filter by eval type (safety, accuracy, jailbreak, image_safety)

statusstring nullable

Filter by eval status (created, processing, finished, failed)

languagestring nullable

Filter by language code (e.g., en, es)

modalitystring nullable

Filter by modality (text, image)

created_bystring nullable

Filter by creator email

created_afterstring date-time nullable

Filter evals created after this date

created_beforestring date-time nullable

Filter evals created before this date

is_jailbreakboolean nullable

Filter by jailbreak status

is_sandboxboolean nullable

Filter by sandbox status

workspace_uuidstring nullable

Filter by workspace UUID

min_pass_ratenumber nullable

Minimum average pass rate (0.0-1.0)

max_pass_ratenumber nullable

Maximum average pass rate (0.0-1.0)

has_score_runsboolean nullable

Only include evals that have score runs

score_run_statusstring nullable

Filter by any score run status

run_created_afterstring date-time nullable

Filter by score runs created after this date

run_created_beforestring date-time nullable

Filter by score runs created before this date

sort_by'created_at' | 'updated_at' | 'name' | 'pass_rate' | 'num_score_runs' | 'last_run_date'

Field to sort by

sort_order'asc' | 'desc'

Sort order

limitinteger

Maximum number of results (1-100)

offsetinteger

Number of results to skip

Response

OK

total_countinteger required

Total number of evals matching the analysis criteria

has_moreboolean required

Whether there are more results available

Changes

Changed in 8 of the 25 revisions of this API.312

  • 0cb64ef11adf2See the full diff
    • ▲

      removed the required property results/items/eval/ai_instructions/anyOf[subschema #2: AgentInstructions]/tools/oneOf[subschema #1: ToolArray]/value/items/id from the response with the 200 status

      response-required-property-removed

    • ▲

      removed the required property results/items/eval/ai_instructions/anyOf[subschema #3: WorkflowInstructions]/instructions/items/tools/oneOf[subschema #1: ToolArray]/value/items/id from the response with the 200 status

      response-required-property-removed

  • cd49adceebf82See the full diff
    • ○

      added subschema #1 subschema #4 to the results/items/eval/ai_instructions response property anyOf list for the response status 200

      response-property-any-of-added

    • ○

      removed subschema #1 subschema #4 from the results/items/eval/ai_instructions response property anyOf list for the response status 200

      response-property-any-of-removed

  • 0b43b7bedf322See the full diff
    • ○

      added the optional property results/items/eval/ai_instructions/anyOf[subschema #2: AgentInstructions]/agent_name to the response with the 200 status

      response-optional-property-added

    • ○

      added WorkflowInstructions to the results/items/eval/ai_instructions response property anyOf list for the response status 200

      response-property-any-of-added

  • 1af31c95d91a12See the full diff
    • ▲

      added ToolArray ToolDict ToolString to the results/items/eval/ai_instructions/anyOf[subschema #2: AgentInstructions]/tools response property oneOf list for the response status 200

      response-property-one-of-added

    • ○

      removed ToolsList ToolDict ToolsString from the results/items/eval/ai_instructions/anyOf[subschema #2: AgentInstructions]/tools response property anyOf list for the response status 200

      response-property-any-of-removed

    • ○

      added discriminator to results/items/eval/ai_instructions/anyOf[subschema #2: AgentInstructions]/tools response property for the response status 200

      response-property-discriminator-added

    This revision also has 2 changes that name no endpoint, such as unreferenced schemas being removed. See the revision's changelog

  • d950d722e2302See the full diff
    • ○

      added subschema #1 subschema #3 to the results/items/eval/ai_instructions/anyOf[subschema #2: AgentInstructions]/tools response property anyOf list for the response status 200

      response-property-any-of-added

    • ○

      removed subschema #1 subschema #3 from the results/items/eval/ai_instructions/anyOf[subschema #2: AgentInstructions]/tools response property anyOf list for the response status 200

      response-property-any-of-removed

  • 33b4b8e9d2ce2See the full diff
    • ○

      added subschema #1 to the results/items/eval/ai_instructions/anyOf[subschema #2: AgentInstructions]/tools response property anyOf list for the response status 200

      response-property-any-of-added

    • ○

      removed subschema #2 from the results/items/eval/ai_instructions/anyOf[subschema #2: AgentInstructions]/tools response property anyOf list for the response status 200

      response-property-any-of-removed

  • b824d75fc3cb1See the full diff
    • ○

      added AgentInstructions to the results/items/eval/ai_instructions response property anyOf list for the response status 200

      response-property-any-of-added

  • 9b529d67a7d01See the full diff
    • ○

      endpoint added

      endpoint-added