Sign in
Eval runs

Create an eval case

Adds one case to the suite. Both title and a non-empty steps array are required — steps is optional on the shared case shape so PATCH can be partial, but a case persisted without executable steps could never run. When models is omitted the suite's configured model is used.

post/projects/{projectId}/eval-suites/{suiteId}/cases

Path parameters

projectIdstring required

ID of the hosted project that contains the server.

suiteIdstring required

Eval suite ID, as returned by POST /eval-runs.

Query parameters

declaredSuiteIdstring

The suite file's own suite.id, when this request is that file syncing itself. A CI-owned suite (see EvalSuiteDetail.managedBy) refuses configuration writes with 409 and details.reason: "CI_OWNED_SUITE_READ_ONLY"; naming the suite's own declared id is how the file writing itself is allowed through. Naming any other id refuses exactly as loudly as naming none, so it is not a capability — omit it for an ordinary edit.

A QUERY PARAMETER on every route that takes it, never a body field. These request bodies are strict, here and on every Inspector that predates the CI-owned lock, so a body field is a 400 against an older deployment — which would break mcpjam cloud eval run --file for anyone whose CLI is newer than their Inspector. A query parameter is read by the deployments that know it and ignored by those that do not, which is the right degradation: an Inspector with no lock has no exception to make.

Request body

declaredSuiteIdstring

The suite file's own suite.id, when THIS request is a suite-file sync.

A CI-owned suite (see EvalSuiteDetail.managedBy) refuses configuration writes with 409 and details.reason: "CI_OWNED_SUITE_READ_ONLY". Naming the suite's own declared id is how the file writing itself is allowed through — the platform permits the write only when the id matches, so naming any other id refuses exactly as loudly as naming none. Omit it for ordinary edits.

PREFER THE declaredSuiteId QUERY PARAMETER on this route. This body field still works here, but these request bodies are strict on every Inspector that predates the CI-owned lock, so a body field is a 400 against an older deployment while an unknown query parameter is simply ignored. The MCPJam SDK and CLI send the query parameter.

titlestring required
idstring

Stable declared id for the case — the id it answers to in a suite file, an import or a CLI argument. Minted for you when omitted, and unique within the suite. Distinct from the id on the response, which is the case's platform row id.

expectedOutputstring
iterationsinteger
repetitionsinteger

Per-case repetitions; omitted cases inherit the suite default. VERDICT POLICY 2 ONLY — on a legacy suite this is a VALIDATION_ERROR naming the upgrade, because the legacy resolver takes the trial count from iterations and the suite's minimumIterations and would never read this. Capped at 10, the same hosted ceiling as iterations; the suite-file contract's larger MAX_REPETITIONS is what a file may DECLARE, not what a hosted run accepts.

passThresholdnumber

Per-case pass threshold fraction. VERDICT POLICY 2 ONLY — on a legacy suite this is a VALIDATION_ERROR naming the upgrade, for the same reason as repetitions.

isNegativeboolean
scenariostring
intentstring

Optional authored analytics grouping label. Must be already trimmed.

kind'capability' | 'regression'

Authored case kind for the simple editor. Absent means the editor derives it from matchOptions.

matchOptionsobject nullable

null clears the case override and inherits the suite's.

suppressedSuiteStandardCheckIdsstring[]

Stable standard assertion family IDs suppressed from suite defaults. Applies before inherit/extend/replace resolution; explicit case and step assertions are preserved. Omitted updates preserve; [] clears. At most 64 IDs, normalized to unique sorted values.

Response

The created case.

idstring required
declaredIdstring

The case's effective declared id. Absent on cases authored before declared identity existed.

titlestring required
expectedOutputstring
iterationsinteger required
repetitionsinteger

Trials this case runs under verdict policy 2, overriding the suite default. Absent means the case inherits it. NOT a second spelling of iterations: that one is the legacy count, which the legacy resolver reads as a FLOOR (max(iterations, suite.minimumIterations)) and which a policy-2 case still reports for compatibility. This one is exact.

passThresholdnumber

Fraction of this case's trials that must pass, overriding the suite default. Absent means the case inherits it. Never derived from the suite's minimumAccuracy, which is a PERCENT under a different resolver.

isNegativeboolean required

When true, the case passes if NO tools are called.

scenariostring
intentstring

Optional authored analytics grouping label. Must be already trimmed; absent means unlabelled.

kind'capability' | 'regression'

Authored case kind for the simple editor. Absent means the editor derives it from matchOptions.

matchOptionsobject

Absent when the case sets none — omitted from the response rather than sent as null.

createdAtnumber nullable
updatedAtnumber nullable
suppressedSuiteStandardCheckIdsstring[]

Stable standard assertion family IDs suppressed from suite defaults. Applies before inherit/extend/replace resolution; explicit case and step assertions are preserved. Omitted updates preserve; [] clears. At most 64 IDs, normalized to unique sorted values.

Changes