Sign in
projects

Create Evaluation Suite

Materialize and pin a project Evaluation Suite from one supported source.

Args: project_id: Owning project. request: Source, canonicalization, and scorer configuration. auth: Authenticated requester.

Returns: Project-owned immutable Evaluation Suite.

Raises: HTTPException: If source cases are invalid or project access is denied.

post/projects/{project_id}/evaluation-suites

Path parameters

project_idstring required

Request body

namestring required
source_kind'agent_generated' | 'huggingface' | 'platform_catalog' | 'upload' required
dataset_idstring nullable
agent_run_idstring nullable
catalog_keystring nullable
task_typestring required
field_mappingobject
scorer_kind'harness' | 'llmaj' | 'native'
scorer_keystring

Reviewed scorer key. Native suites accept native.v1, plus exact_match for decoder tasks; unsupported native task/key pairs are rejected. LLMAJ and harness scorers retain their own reviewed string namespaces, so this field cannot be a global enum.

max_casesinteger nullable

Response

Successful Response

idstring required
project_idstring required
dataset_idstring required
namestring required
task_typestring required
source_kind'agent_generated' | 'huggingface' | 'platform_catalog' | 'upload' required
source_provenanceJsonObjectOutput required
field_mappingJsonObjectOutput required
scorer_kind'harness' | 'llmaj' | 'native' required
scorer_configJsonObjectOutput required
case_countinteger required
created_atstring date-time required
updated_atstring date-time required

Changes