Eval runs

Get the pre-run disclosure for a suite launch plan

What happens to a run's content BEFORE you launch it: which models it calls and where they route, which LLM analyzers/judges can fire and where their evidence goes, capture/retention/region facts, and the subprocessors engaged. Read-only — never launches or gates a run. Keyed by the SAME destination-affecting subset a launch selects (caseIds/environmentId/environmentIds); pass the selectors you would pass to a run so what this discloses is what that run would do. run_eval_suite already fetches this itself and returns it on the receipt's disclosure field — call this route directly when you need the disclosure BEFORE deciding to launch. A deployment that predates this contract answers 422 FEATURE_NOT_SUPPORTED with details.reason: "contract_unavailable", never a partial payload — this is a GUARANTEE only when the deployment's missing-function error reaches this route unredacted. Production Convex can redact that same failure to a generic "Server Error" indistinguishable from a genuine handler crash on a suite the caller CAN see; this route has no independent way to tell the two apart in that one case, so it answers 502 instead of guessing 422 — a caller cannot rely on this code alone to detect an old production deployment, and a 502 does not imply the contract is available either. projectId is positional only, for REST consistency with sibling routes — this endpoint authorizes per-suite, not per-project, so any projectId returns the same disclosure for a suiteId you can reach.

get/projects/{projectId}/eval-suites/{suiteId}/run-disclosure

Path parameters

projectIdstring required

ID of the hosted project that contains the server.

suiteIdstring required

Eval suite ID, as returned by POST /eval-runs.

Query parameters

caseIdsstring

Comma-separated test case IDs to narrow the disclosure to, instead of the whole suite.

environmentIdstring

One attached environment to disclose for. Mutually exclusive with environmentIds and host.

environmentIdsstring

Comma-separated attached environment IDs to disclose for, mirroring a multi-target launch. Mutually exclusive with environmentId and host.

hoststring

One attached host to disclose for, mirroring a host-targeted launch — the engine and sandbox facts come from that host's own config. Mutually exclusive with environmentId/environmentIds: a launch plan resolves on exactly one axis.

Headers

x-mcpjam-eval-vocabulary'1' | '2'

Which vocabulary this request and its response speak. Absent means 1, which is byte-for-byte today's contract: the same request fields, the same refusals, the same response projection. 2 is the canonical vocabulary. Any other value is a 400 with code: "VALIDATION_ERROR".

Today it decides one thing: the spelling of an evaluator's policy role. Vocabulary 1 accepts and returns gating; vocabulary 2 accepts both spellings and returns the canonical required. Sending required without the header is a 400, deliberately — vocabulary 1 is not widened to meet vocabulary 2 half way, because a boundary that accepts a spelling it does not announce is one two implementations can disagree about.

A response that varies by vocabulary sends Vary: x-mcpjam-eval-vocabulary.

Response

The pre-run disclosure.

contractVersioninteger required
computedAtinteger required

Epoch ms. Excluded from digest, so identical facts digest identically regardless of when they were computed.

digeststring required

SHA-256 hex over the canonical JSON of the facts (excluding digest/computedAt and each managed rail's observedAt).

analysisobject[] required
captureobject required

Capture level, reporting mode, and redaction facts (fixed for this contract version). redaction.providerRetention carries the provider policy every platform-key analysis call sends (zero data retention and no training, each read off the wire flags) and names what it does not cover; it is absent on older backends.

retentionobject required

Plan retention policy, whether it is actually enforced, and what that means today (kept-indefinitely vs swept-after-policy-days).

regionobject required

{ stated: false, reason } unless a BYOK base URL carries a derivable region token.

subprocessorsobject[] required

Changes