Sign in
projects

Start Evaluation Suite Runs

Create append-only Evaluation Suite runs and dispatch existing evaluators.

Args: project_id: Owning project. suite_id: Pinned Evaluation Suite to execute. request: Project models and optional case cap. http_request: Incoming request, used to resolve a run key's approved plan (ENG-7005). auth: Authenticated requester.

Returns: Pending run records, one for each selected model.

Raises: HTTPException: If suite/model ownership or scorer compatibility is invalid.

post/projects/{project_id}/evaluation-suites/{suite_id}/runs

Path parameters

project_idstring required
suite_idstring required

Request body

model_idsstring[] required
max_examplesinteger nullable

Optional execution cap. Omit to run every pinned case. Reviewed catalogue splits exceed 10,000 cases, so this field has no upper bound; dispatch still refuses unbounded model fan-out.

cls_thresholdnumber nullable

Optional decision threshold for the multi-label classification head (GLiNER2 cls_threshold). Only valid for classification Evaluation Suites; rejected for every other task_type. Must be in [0.0, 1.0] inclusive, matching coerce_cls_threshold's own range check -- 0.0 returns every label, 1.0 falls back to argmax, both meaningful.

sweep_cls_thresholdsnumber[] nullable

Optional multi-label classification decision thresholds to score for free from this run's own inference pass (ENG-6584). Only valid for classification Evaluation Suites; rejected for every other task_type. Each candidate must be >= 0.5, the floor every inference call runs at today, and <= 1.0. Results land in the completed run's own metrics as 'threshold_sweep' and 'threshold_sweep_best'. Independent of cls_threshold: that field pins the single threshold this run is scored and recorded at, this field additionally sweeps a grid of candidates from the same inference pass -- use it to find a winner cheaply, then a second run with cls_threshold=<winner> to record it durably.

experiment_idstring nullable

Project-scoped experiment requesting this run.

Response

Successful Response

idstring required
suite_idstring required
project_idstring required
model_idstring required
status'awaiting_scoring' | 'cancelled' | 'complete' | 'failed' | 'pending' | 'running' required
max_examplesinteger nullable required
scorer_snapshotJsonObjectOutput required
comparison_groupstring nullable required

Scorer id, version, and behaviour digest. Runs without the same non-null value must not be ranked or compared.

metricsJsonObjectOutput required
error_messagestring nullable required
created_atstring date-time required
started_atstring date-time nullable required
lease_expires_atstring date-time nullable required
scoring_expires_atstring date-time nullable required
completed_atstring date-time nullable required

Changes