PIpioneer
projects

Start Evaluation Suite Runs

Changed on

Create append-only Evaluation Suite runs for agent-sandbox execution.

Args: project_id: Owning project. suite_id: Pinned Evaluation Suite to execute. request: Project models and optional case cap. http_request: Incoming request, used to resolve a run key's approved plan (ENG-7005). auth: Authenticated requester.

Returns: Prepared run records, one for each selected model.

Raises: HTTPException: If suite/model ownership or scorer compatibility is invalid.

post/v1/projects/{project_id}/evaluation-suites/{suite_id}/runs

Request

Path parameters

project_idstring required
suite_idstring required

Request body

model_idsstring[] required
num_drawsinteger

Independent full-suite sandbox inference passes. One preserves the current cost profile; repeated draws persist mean and sample standard deviation alongside the usual scalar metrics.

max_examplesinteger nullable

Optional execution cap. Omit to run every pinned case. Reviewed catalogue splits exceed 10,000 cases, so this field has no upper bound; dispatch still refuses unbounded model fan-out.

cls_thresholdnumber nullable

Optional decision threshold for the multi-label classification head (GLiNER2 cls_threshold). Only valid for classification Evaluation Suites; rejected for every other task_type. Must be in [0.0, 1.0] inclusive, matching coerce_cls_threshold's own range check -- 0.0 returns every label, 1.0 falls back to argmax, both meaningful.

experiment_idstring nullable

Project-scoped experiment requesting this run.

Response

Successful Response

idstring required
suite_idstring required
project_idstring required
model_idstring required
status'awaiting_scoring' | 'cancelled' | 'complete' | 'failed' | 'pending' | 'running' required
max_examplesinteger nullable required
scorer_snapshotJsonObjectOutput required
comparison_groupstring nullable required

Scorer id, version, and behaviour digest. Runs without the same non-null value must not be ranked or compared.

metricsJsonObjectOutput required
error_messagestring nullable required
created_atstring date-time required
started_atstring date-time nullable required
lease_expires_atstring date-time nullable required
scoring_expires_atstring date-time nullable required
completed_atstring date-time nullable required

Changes