Eval runs

Get one eval case

Changed on

A case that belongs to this suite and project. A case in another suite is a 404, not a 403.

get/projects/{projectId}/eval-suites/{suiteId}/cases/{caseId}

Request

  • Base URL
    https://app.mcpjam.com/api/v1
  • URL
    https://app.mcpjam.com/api/v1/projects/{projectId}/eval-suites/{suiteId}/cases/{caseId}
  • Auth
    HTTP bearer

Path parameters

projectIdstring required

ID of the hosted project that contains the server.

suiteIdstring required

Eval suite ID, as returned by POST /eval-runs.

caseIdstring required

Eval case id.

Headers

x-mcpjam-eval-vocabulary'1' | '2'

Which vocabulary this request and its response speak. Absent means 1, which is byte-for-byte today's contract: the same request fields, the same refusals, the same response projection. 2 is the canonical vocabulary. Any other value is a 400 with code: "VALIDATION_ERROR".

Today it decides one thing: the spelling of an evaluator's policy role. Vocabulary 1 accepts and returns gating; vocabulary 2 accepts both spellings and returns the canonical required. Sending required without the header is a 400, deliberately — vocabulary 1 is not widened to meet vocabulary 2 half way, because a boundary that accepts a spelling it does not announce is one two implementations can disagree about.

A response that varies by vocabulary sends Vary: x-mcpjam-eval-vocabulary.

Response

The case.

idstring required
declaredIdstring

The case's effective declared id. Absent on cases authored before declared identity existed.

titlestring required
expectedOutputstring
iterationsinteger required
repetitionsinteger

Trials this case runs under per-case grading, overriding the suite default. Absent means the case inherits it. NOT a second spelling of iterations: that one is read as a FLOOR (max(iterations, suite.minimumIterations)) by a suite decided on suite-wide accuracy, and a per-case-graded case still reports it for compatibility. This one REPLACES the count rather than raising it — a case at 7 runs 7 times under a floor of 3 and 3 times under a default of 3.

passThresholdnumber

Fraction of this case's trials that must pass, overriding the suite default. Absent means the case inherits it. Never derived from the suite's minimumAccuracy, which is a PERCENT under a different resolver.

isNegativeboolean required

When true, the case passes if NO tools are called.

scenariostring
intentstring

Optional authored analytics grouping label. Must be already trimmed; absent means unlabelled.

kind'capability' | 'regression'

Authored case kind for the simple editor. Absent means the editor derives it from matchOptions.

matchOptionsobject

Absent when the case sets none — omitted from the response rather than sent as null.

createdAtnumber nullable
updatedAtnumber nullable
suppressedSuiteStandardCheckIdsstring[]

Stable standard assertion family IDs suppressed from suite defaults. Applies before inherit/extend/replace resolution; explicit case and step assertions are preserved. Omitted updates preserve; [] clears. At most 64 IDs, normalized to unique sorted values.

Changes

  • 9826d734d268Revision changes
    • info

      added the optional property // to the response with the status

    • info

      added to the // response property allOf list for the response status

    • info

      removed from the // response property allOf list for the response status

    • info

      the response required property // became read-only for the status

  • c9e609871357Revision changes
    • info

      added the optional property // to the response with the status