Sign in
Eval runs

Generate eval cases from the suite's tools

Discovers the suite's server tools over a live MCP connection, generates cases against them, and persists them — the only edit route that connects to a server. Synchronous: connect, generate, persist, disconnect, respond.

The authoring model is on MCPJam: no customer credits are consumed. The call still counts against the organization's daily generation quota, and against MCPJam's own daily budget for generation, so it can be refused with 429 RATE_LIMITED and a Retry-After. details.canTopUp is false: wait until Retry-After rather than topping up or retrying in a loop.

An environment-based suite generates against that environment's closed server set, so the cases match the tools its runs will actually see.

Pass x-mcpjam-idempotency-key to make a retry safe: drafts are recorded backend-side BEFORE any case is persisted, so a replay reuses them instead of taking a second slice of the daily quota, and each case is persisted under a derived per-item key so the loop is resumable.

post/projects/{projectId}/eval-suites/{suiteId}/cases/generate

Path parameters

projectIdstring required

ID of the hosted project that contains the server.

suiteIdstring required

Eval suite ID, as returned by POST /eval-runs.

Headers

x-mcpjam-eval-vocabulary'1' | '2'

Which vocabulary this request and its response speak. Absent means 1, which is byte-for-byte today's contract: the same request fields, the same refusals, the same response projection. 2 is the canonical vocabulary. Any other value is a 400 with code: "VALIDATION_ERROR".

Today it decides one thing: the spelling of an evaluator's policy role. Vocabulary 1 accepts and returns gating; vocabulary 2 accepts both spellings and returns the canonical required. Sending required without the header is a 400, deliberately — vocabulary 1 is not widened to meet vocabulary 2 half way, because a boundary that accepts a spelling it does not announce is one two implementations can disagree about.

A response that varies by vocabulary sends Vary: x-mcpjam-eval-vocabulary.

x-mcpjam-idempotency-keystring

Makes a retry replay the recorded drafts instead of taking a second slice of the organization's daily generation quota.

Request body

mode'normal' | 'negative'

Superseded by caseMix when that is present.

serversstring[]

Server ids or names to discover tools from. Ignored when the suite is environment-based.

environmentIdstring

Discover tools from this attached environment's closed server set, so generated cases are written against the tools the suite's runs will actually see.

varyUserStylesboolean

Condition generated cases on a range of user styles so the queries read like different users wrote them.

idempotencyKeystring

Write-idempotency key. A repeat call with the same key replays recorded drafts instead of spending credits again. The Idempotency-Key / x-mcpjam-idempotency-key header carries the same value and WINS over this field.

Response

The generated cases.

generationModelstring required
skippedobject[]

Drafts that were generated but failed to persist. Surfaced rather than silently dropped.

Changes

Changed in 12 of the 122 revisions of this API.314

  • 311db07449791See the full diff
    • ○

      added the success response with the status

      response-success-status-added

  • 0e05c5e4277f1See the full diff
    • ○

      added the optional property // to the response with the status

      response-optional-property-added

  • 56b9d1dda7ea12See the full diff
    • ▲

      added subschema #1 subschema #2 subschema #3 subschema #4 subschema #5 subschema #6 subschema #7 subschema #8 subschema #9 subschema #10 subschema #11 subschema #12 subschema #13 subschema #14 subschema #15 subschema #16 subschema #17 subschema #18 subschema #19 subschema #20 subschema #21 subschema #22 subschema #23 subschema #24 subschema #25 subschema #26 subschema #27 subschema #28 subschema #29 subschema #30 subschema #31 subschema #32 subschema #33 to the //// response property oneOf list for the response status

      response-property-one-of-added

    • ○

      added the new optional header request parameter x-mcpjam-eval-vocabulary

      new-optional-request-parameter

    • ○

      removed subschema #1 subschema #2 subschema #3 subschema #4 subschema #5 subschema #6 subschema #7 subschema #8 subschema #9 subschema #10 subschema #11 subschema #12 subschema #13 subschema #14 subschema #15 subschema #16 subschema #17 subschema #18 subschema #19 subschema #20 subschema #21 subschema #22 subschema #23 subschema #24 subschema #25 subschema #26 subschema #27 subschema #28 subschema #29 subschema #30 subschema #31 subschema #32 subschema #33 from the //// response property oneOf list for the response status

      response-property-one-of-removed

  • 16cc71fe1f3321See the full diff
    • ▲

      added subschema #1 subschema #2 subschema #3 subschema #4 subschema #5 subschema #6 subschema #7 subschema #8 subschema #9 subschema #10 subschema #11 subschema #12 subschema #13 subschema #14 subschema #15 subschema #16 subschema #17 subschema #18 subschema #19 subschema #20 subschema #21 subschema #22 subschema #23 subschema #24 subschema #25 subschema #26 subschema #27 subschema #28 subschema #29 subschema #30 subschema #31 subschema #32 subschema #33 to the //// response property oneOf list for the response status

      response-property-one-of-added

    • ▲

      the //// response's property type changed from object to no type for status

      response-property-type-changed

    • ○

      added the optional property // to the response with the status

      response-optional-property-added

  • cec7381898e21See the full diff
    • ○

      added the optional property // to the response with the status

      response-optional-property-added

  • 7584cc19e5ff1See the full diff
    • ○

      added the optional property // to the response with the status

      response-optional-property-added

  • d9a00becd4761See the full diff
    • ○

      added the optional property // to the response with the status

      response-optional-property-added

  • e81e8ba8baae1See the full diff
    • ○

      added the optional property // to the response with the status

      response-optional-property-added

  • c479eeceb7a32See the full diff
    • ○

      added the optional property // to the response with the status

      response-optional-property-added

    • ○

      added the optional property // to the response with the status

      response-optional-property-added

  • a0fa11827b5e1See the full diff
    • ○

      added the new optional request property

      new-optional-request-property