Sign in
Eval runs

Create an eval case

Adds one case to the suite. Both title and a non-empty steps array are required — steps is optional on the shared case shape so PATCH can be partial, but a case persisted without executable steps could never run. When models is omitted the suite's configured model is used.

post/projects/{projectId}/eval-suites/{suiteId}/cases

Path parameters

projectIdstring required

ID of the hosted project that contains the server.

suiteIdstring required

Eval suite ID, as returned by POST /eval-runs.

Query parameters

declaredSuiteIdstring

The suite file's own suite.id, when this request is that file syncing itself. A CI-owned suite (see EvalSuiteDetail.managedBy) refuses configuration writes with 409 and details.reason: "CI_OWNED_SUITE_READ_ONLY"; naming the suite's own declared id is how the file writing itself is allowed through. Naming any other id refuses exactly as loudly as naming none, so it is not a capability — omit it for an ordinary edit.

A QUERY PARAMETER on every route that takes it, never a body field. These request bodies are strict, here and on every Inspector that predates the CI-owned lock, so a body field is a 400 against an older deployment — which would break mcpjam cloud eval run --file for anyone whose CLI is newer than their Inspector. A query parameter is read by the deployments that know it and ignored by those that do not, which is the right degradation: an Inspector with no lock has no exception to make.

Headers

x-mcpjam-eval-vocabulary'1' | '2'

Which vocabulary this request and its response speak. Absent means 1, which is byte-for-byte today's contract: the same request fields, the same refusals, the same response projection. 2 is the canonical vocabulary. Any other value is a 400 with code: "VALIDATION_ERROR".

Today it decides one thing: the spelling of an evaluator's policy role. Vocabulary 1 accepts and returns gating; vocabulary 2 accepts both spellings and returns the canonical required. Sending required without the header is a 400, deliberately — vocabulary 1 is not widened to meet vocabulary 2 half way, because a boundary that accepts a spelling it does not announce is one two implementations can disagree about.

A response that varies by vocabulary sends Vary: x-mcpjam-eval-vocabulary.

Request body

declaredSuiteIdstring

The suite file's own suite.id, when THIS request is a suite-file sync.

A CI-owned suite (see EvalSuiteDetail.managedBy) refuses configuration writes with 409 and details.reason: "CI_OWNED_SUITE_READ_ONLY". Naming the suite's own declared id is how the file writing itself is allowed through — the platform permits the write only when the id matches, so naming any other id refuses exactly as loudly as naming none. Omit it for ordinary edits.

PREFER THE declaredSuiteId QUERY PARAMETER on this route. This body field still works here, but these request bodies are strict on every Inspector that predates the CI-owned lock, so a body field is a 400 against an older deployment while an unknown query parameter is simply ignored. The MCPJam SDK and CLI send the query parameter.

titlestring required
idstring

Stable declared id for the case — the id it answers to in a suite file, an import or a CLI argument. Minted for you when omitted, and unique within the suite. Distinct from the id on the response, which is the case's platform row id.

expectedOutputstring
iterationsinteger
repetitionsinteger

Per-case repetitions; omitted cases inherit the suite default. PER-CASE GRADING ONLY — on a suite decided by the suite-wide accuracy threshold this is a VALIDATION_ERROR naming the criterion change that would make it readable, because that criterion takes the trial count from iterations raised to the suite's minimumIterations and would never read this. Capped at 10, the same hosted ceiling as iterations; the suite-file contract's larger MAX_REPETITIONS is what a file may DECLARE, not what a hosted run accepts.

passThresholdnumber

Per-case pass threshold FRACTION: the share of this case's own iterations that must pass. PER-CASE GRADING ONLY — on a suite decided by the suite-wide accuracy threshold this is a VALIDATION_ERROR, for the same reason as the count: that criterion has one percentage over the whole run and no per-case threshold to override.

isNegativeboolean
scenariostring
intentstring

Optional authored analytics grouping label. Must be already trimmed.

kind'capability' | 'regression'

Authored case kind for the simple editor. Absent means the editor derives it from matchOptions.

matchOptionsobject nullable

null clears the case override and inherits the suite's.

suppressedSuiteStandardCheckIdsstring[]

Stable standard assertion family IDs suppressed from suite defaults. Applies before inherit/extend/replace resolution; explicit case and step assertions are preserved. Omitted updates preserve; [] clears. At most 64 IDs, normalized to unique sorted values.

Response

The created case.

idstring required
declaredIdstring

The case's effective declared id. Absent on cases authored before declared identity existed.

titlestring required
expectedOutputstring
iterationsinteger required
repetitionsinteger

Trials this case runs under per-case grading, overriding the suite default. Absent means the case inherits it. NOT a second spelling of iterations: that one is read as a FLOOR (max(iterations, suite.minimumIterations)) by a suite decided on suite-wide accuracy, and a per-case-graded case still reports it for compatibility. This one REPLACES the count rather than raising it — a case at 7 runs 7 times under a floor of 3 and 3 times under a default of 3.

passThresholdnumber

Fraction of this case's trials that must pass, overriding the suite default. Absent means the case inherits it. Never derived from the suite's minimumAccuracy, which is a PERCENT under a different resolver.

isNegativeboolean required

When true, the case passes if NO tools are called.

scenariostring
intentstring

Optional authored analytics grouping label. Must be already trimmed; absent means unlabelled.

kind'capability' | 'regression'

Authored case kind for the simple editor. Absent means the editor derives it from matchOptions.

matchOptionsobject

Absent when the case sets none — omitted from the response rather than sent as null.

createdAtnumber nullable
updatedAtnumber nullable
suppressedSuiteStandardCheckIdsstring[]

Stable standard assertion family IDs suppressed from suite defaults. Applies before inherit/extend/replace resolution; explicit case and step assertions are preserved. Omitted updates preserve; [] clears. At most 64 IDs, normalized to unique sorted values.

Changes