AYaymara-ai
Eval Analysis

Analyze Evals

Changed on

Analysis for evals with advanced filtering and aggregated statistics.

This endpoint allows analyzing across both eval metadata and score run performance data, providing comprehensive filtering capabilities and aggregated statistics for each eval.

Args: analysis_request (EvalAnalysisRequest): Analysis parameters and filters including: - Eval metadata filters (name, type, status, language, etc.) - Score run performance filters (pass rate, run count, etc.) - Sorting and pagination options

Returns: EvalAnalysisResponse: Paginated results with matching evals and their statistics

Raises: AymaraAPIError: If the request is invalid or analysis parameters are malformed

Example: POST /api/v2/eval_analysis { "name": "safety", "eval_type": "safety", "min_pass_rate": 0.8, "has_score_runs": true, "sort_by": "pass_rate", "sort_order": "desc", "limit": 20, "offset": 0 }

post/v2/eval-analysis

Request

Request body

namestring nullable

Filter by eval names (case-insensitive partial match)

eval_typestring nullable

Filter by eval type (safety, accuracy, jailbreak, image_safety)

statusstring nullable

Filter by eval status (created, processing, finished, failed)

languagestring nullable

Filter by language code (e.g., en, es)

modalitystring nullable

Filter by modality (text, image)

created_bystring nullable

Filter by creator email

created_afterstring date-time nullable

Filter evals created after this date

created_beforestring date-time nullable

Filter evals created before this date

is_jailbreakboolean nullable

Filter by jailbreak status

is_sandboxboolean nullable

Filter by sandbox status

workspace_uuidstring nullable

Filter by workspace UUID

min_pass_ratenumber nullable

Minimum average pass rate (0.0-1.0)

max_pass_ratenumber nullable

Maximum average pass rate (0.0-1.0)

has_score_runsboolean nullable

Only include evals that have score runs

score_run_statusstring nullable

Filter by any score run status

run_created_afterstring date-time nullable

Filter by score runs created after this date

run_created_beforestring date-time nullable

Filter by score runs created before this date

sort_by'created_at' | 'updated_at' | 'name' | 'pass_rate' | 'num_score_runs' | 'last_run_date'

Field to sort by

sort_order'asc' | 'desc'

Sort order

limitinteger

Maximum number of results (1-100)

offsetinteger

Number of results to skip

Response

OK

total_countinteger required

Total number of evals matching the analysis criteria

has_moreboolean required

Whether there are more results available

Changes

  • 0cb64ef11adfRevision changes
    • breaking

      removed the required property ///////// from the response with the status

    • breaking

      removed the required property /////////// from the response with the status

  • cd49adceebf8Revision changes
    • info

      added subschema #1 subschema #4 to the /// response property anyOf list for the response status

    • info

      removed subschema #1 subschema #4 from the /// response property anyOf list for the response status