Sign in
Eval runs

Update an eval case

Partial update — omitted fields are left as they are. Supplying steps REPLACES the case's definition rather than merging into it. matchOptions and checks accept null to clear the case-level override and fall back to the suite's.

patch/projects/{projectId}/eval-suites/{suiteId}/cases/{caseId}

Path parameters

projectIdstring required

ID of the hosted project that contains the server.

suiteIdstring required

Eval suite ID, as returned by POST /eval-runs.

caseIdstring required

Eval case id.

Query parameters

declaredSuiteIdstring

The suite file's own suite.id, when this request is that file syncing itself. A CI-owned suite (see EvalSuiteDetail.managedBy) refuses configuration writes with 409 and details.reason: "CI_OWNED_SUITE_READ_ONLY"; naming the suite's own declared id is how the file writing itself is allowed through. Naming any other id refuses exactly as loudly as naming none, so it is not a capability — omit it for an ordinary edit.

A QUERY PARAMETER on every route that takes it, never a body field. These request bodies are strict, here and on every Inspector that predates the CI-owned lock, so a body field is a 400 against an older deployment — which would break mcpjam cloud eval run --file for anyone whose CLI is newer than their Inspector. A query parameter is read by the deployments that know it and ignored by those that do not, which is the right degradation: an Inspector with no lock has no exception to make.

Headers

x-mcpjam-eval-vocabulary'1' | '2'

Which vocabulary this request and its response speak. Absent means 1, which is byte-for-byte today's contract: the same request fields, the same refusals, the same response projection. 2 is the canonical vocabulary. Any other value is a 400 with code: "VALIDATION_ERROR".

Today it decides one thing: the spelling of an evaluator's policy role. Vocabulary 1 accepts and returns gating; vocabulary 2 accepts both spellings and returns the canonical required. Sending required without the header is a 400, deliberately — vocabulary 1 is not widened to meet vocabulary 2 half way, because a boundary that accepts a spelling it does not announce is one two implementations can disagree about.

A response that varies by vocabulary sends Vary: x-mcpjam-eval-vocabulary.

Request body

declaredSuiteIdstring

The suite file's own suite.id, when THIS request is a suite-file sync.

A CI-owned suite (see EvalSuiteDetail.managedBy) refuses configuration writes with 409 and details.reason: "CI_OWNED_SUITE_READ_ONLY". Naming the suite's own declared id is how the file writing itself is allowed through — the platform permits the write only when the id matches, so naming any other id refuses exactly as loudly as naming none. Omit it for ordinary edits.

PREFER THE declaredSuiteId QUERY PARAMETER on this route. This body field still works here, but these request bodies are strict on every Inspector that predates the CI-owned lock, so a body field is a 400 against an older deployment while an unknown query parameter is simply ignored. The MCPJam SDK and CLI send the query parameter.

titlestring
expectedOutputstring
iterationsinteger
repetitionsinteger

Per-case repetitions; omitted cases inherit the suite default. VERDICT POLICY 2 ONLY — on a legacy suite this is a VALIDATION_ERROR naming the upgrade, because the legacy resolver takes the trial count from iterations and the suite's minimumIterations and would never read this. Capped at 10, the same hosted ceiling as iterations; the suite-file contract's larger MAX_REPETITIONS is what a file may DECLARE, not what a hosted run accepts.

passThresholdnumber

Per-case pass threshold fraction. VERDICT POLICY 2 ONLY — on a legacy suite this is a VALIDATION_ERROR naming the upgrade, for the same reason as repetitions.

isNegativeboolean
scenariostring
intentstring nullable

Optional authored analytics grouping label. Must be already trimmed; omitted leaves the stored value unchanged and null clears it.

kind'capability' | 'regression' | 'null' nullable

Authored case kind. Omitted leaves the stored value unchanged; null clears it.

matchOptionsobject nullable

null clears the case override and inherits the suite's.

suppressedSuiteStandardCheckIdsstring[]

Stable standard assertion family IDs suppressed from suite defaults. Applies before inherit/extend/replace resolution; explicit case and step assertions are preserved. Omitted updates preserve; [] clears. At most 64 IDs, normalized to unique sorted values.

Response

The updated case.

idstring required
declaredIdstring

The case's effective declared id. Absent on cases authored before declared identity existed.

titlestring required
expectedOutputstring
iterationsinteger required
repetitionsinteger

Trials this case runs under verdict policy 2, overriding the suite default. Absent means the case inherits it. NOT a second spelling of iterations: that one is the legacy count, which the legacy resolver reads as a FLOOR (max(iterations, suite.minimumIterations)) and which a policy-2 case still reports for compatibility. This one is exact.

passThresholdnumber

Fraction of this case's trials that must pass, overriding the suite default. Absent means the case inherits it. Never derived from the suite's minimumAccuracy, which is a PERCENT under a different resolver.

isNegativeboolean required

When true, the case passes if NO tools are called.

scenariostring
intentstring

Optional authored analytics grouping label. Must be already trimmed; absent means unlabelled.

kind'capability' | 'regression'

Authored case kind for the simple editor. Absent means the editor derives it from matchOptions.

matchOptionsobject

Absent when the case sets none — omitted from the response rather than sent as null.

createdAtnumber nullable
updatedAtnumber nullable
suppressedSuiteStandardCheckIdsstring[]

Stable standard assertion family IDs suppressed from suite defaults. Applies before inherit/extend/replace resolution; explicit case and step assertions are preserved. Omitted updates preserve; [] clears. At most 64 IDs, normalized to unique sorted values.

Changes