Sign in
Eval runs

Generate eval cases from the suite's tools

Discovers the suite's server tools over a live MCP connection, generates cases against them, and persists them — the only edit route that connects to a server, and the only one that SPENDS ORG CREDITS. Synchronous: connect, generate, persist, disconnect, respond.

An environment-based suite generates against that environment's closed server set, so the cases match the tools its runs will actually see.

Pass x-mcpjam-idempotency-key to make a retry safe: drafts are recorded backend-side BEFORE any case is persisted, so a replay reuses them instead of spending credits again, and each case is persisted under a derived per-item key so the loop is resumable.

post/projects/{projectId}/eval-suites/{suiteId}/cases/generate

Path parameters

projectIdstring required

ID of the hosted project that contains the server.

suiteIdstring required

Eval suite ID, as returned by POST /eval-runs.

Headers

x-mcpjam-eval-vocabulary'1' | '2'

Which vocabulary this request and its response speak. Absent means 1, which is byte-for-byte today's contract: the same request fields, the same refusals, the same response projection. 2 is the canonical vocabulary. Any other value is a 400 with code: "VALIDATION_ERROR".

Today it decides one thing: the spelling of an evaluator's policy role. Vocabulary 1 accepts and returns gating; vocabulary 2 accepts both spellings and returns the canonical required. Sending required without the header is a 400, deliberately — vocabulary 1 is not widened to meet vocabulary 2 half way, because a boundary that accepts a spelling it does not announce is one two implementations can disagree about.

A response that varies by vocabulary sends Vary: x-mcpjam-eval-vocabulary.

x-mcpjam-idempotency-keystring

Makes a retry replay the recorded drafts instead of spending credits again.

Request body

mode'normal' | 'negative'

Superseded by caseMix when that is present.

serversstring[]

Server ids or names to discover tools from. Ignored when the suite is environment-based.

environmentIdstring

Discover tools from this attached environment's closed server set, so generated cases are written against the tools the suite's runs will actually see.

varyUserStylesboolean

Condition generated cases on a range of user styles so the queries read like different users wrote them.

idempotencyKeystring

Write-idempotency key. A repeat call with the same key replays recorded drafts instead of spending credits again. The Idempotency-Key / x-mcpjam-idempotency-key header carries the same value and WINS over this field.

Response

The generated cases.

generationModelstring required
skippedobject[]

Drafts that were generated but failed to persist. Surfaced rather than silently dropped.

Changes