--- title: "Generate eval cases from the suite's tools" method: POST path: "/projects/{projectId}/eval-suites/{suiteId}/cases/generate" tags: ["Eval runs"] --- # Generate eval cases from the suite's tools `POST /projects/{projectId}/eval-suites/{suiteId}/cases/generate` Discovers the suite's server tools over a live MCP connection, generates cases against them, and persists them — the only edit route that connects to a server, and the only one that SPENDS ORG CREDITS. Synchronous: connect, generate, persist, disconnect, respond. An environment-based suite generates against that environment's closed server set, so the cases match the tools its runs will actually see. Pass `x-mcpjam-idempotency-key` to make a retry safe: drafts are recorded backend-side BEFORE any case is persisted, so a replay reuses them instead of spending credits again, and each case is persisted under a derived per-item key so the loop is resumable. ## Path parameters - `projectId` string, required - `suiteId` string, required ## Headers - `x-mcpjam-eval-vocabulary` '1' | '2' - `x-mcpjam-idempotency-key` string ## Request body - EvalCaseGenerateRequest — AI-generate cases from the suite's server tools and persist them. SPENDS ORG CREDITS. - `mode` 'normal' | 'negative' — Superseded by `caseMix` when that is present. - `servers` string[] — Server ids or names to discover tools from. Ignored when the suite is environment-based. - `environmentId` string — Discover tools from this attached environment's closed server set, so generated cases are written against the tools the suite's runs will actually see. - `caseModels` object[] - `model` string, required - `provider` string - `caseMix` object — Per-bucket case counts. Omitted buckets inherit the default mix; the backend bounds each bucket and the total. - `simple` integer - `multiTool` integer - `multiTurn` integer - `complex` integer - `negative` integer - `varyUserStyles` boolean — Condition generated cases on a range of user styles so the queries read like different users wrote them. - `idempotencyKey` string — Write-idempotency key. A repeat call with the same key replays recorded drafts instead of spending credits again. The `Idempotency-Key` / `x-mcpjam-idempotency-key` header carries the same value and WINS over this field. ## Response `200` The generated cases. - EvalCaseGenerated - `generationModel` string, required - `created` EvalCase[], required - `judge` object — Per-case judge override. enabled: false skips goal-completion grading for this case. - `enabled` boolean - `id` string, required - `declaredId` string — The case's effective declared id. Absent on cases authored before declared identity existed. - `title` string, required - `steps` EvalTestStep[], required — Ordered test steps. A `prompt` step is a model turn; a single model-free `toolCall` step is a render-check; `assert` steps hold the expectations. - `id` string, required - `kind` 'prompt' | 'toolCall' | 'interact' | 'assert', required - `prompt` string — User message (`kind: prompt`). - `serverName` string — Server that owns the tool (`kind: toolCall`). - `toolName` string — Tool name (`kind: toolCall` / `interact`). - `arguments` object — Tool-call arguments (`kind: toolCall`). - `action` object — Widget action (`kind: interact`). - `assertion` object — The rule an `assert` step checks (`kind: assert`). NOT a narrower spelling of a case's `checks`: this is `WidgetAssertion | Predicate`, and that union is why the field is called an assertion rather than a check. A `Predicate` is evaluated against the PERSISTED transcript, so a stored run can be re-graded against it months later; a `WidgetAssertion` is evaluated against a LIVE DOM and can never be replayed. Calling this a check would promise the replayability only one half of it has. - `expectedOutput` string - `iterations` integer, required - `repetitions` integer — Trials this case runs under per-case grading, overriding the suite default. Absent means the case inherits it. NOT a second spelling of `iterations`: that one is read as a FLOOR (`max(iterations, suite.minimumIterations)`) by a suite decided on suite-wide accuracy, and a per-case-graded case still reports it for compatibility. This one REPLACES the count rather than raising it — a case at 7 runs 7 times under a floor of 3 and 3 times under a default of 3. - `passThreshold` number — Fraction of this case's trials that must pass, overriding the suite default. Absent means the case inherits it. Never derived from the suite's `minimumAccuracy`, which is a PERCENT under a different resolver. - `isNegative` boolean, required — When true, the case passes if NO tools are called. - `scenario` string - `intent` string — Optional authored analytics grouping label. Must be already trimmed; absent means unlabelled. - `kind` 'capability' | 'regression' — Authored case kind for the simple editor. Absent means the editor derives it from matchOptions. - `models` object[], required - `model` string, required - `provider` string - `matchOptions` object — Absent when the case sets none — omitted from the response rather than sent as `null`. - `checks` object — Absent when the case sets none — omitted from the response rather than sent as `null`. - `mode` 'inherit' | 'replace' | 'extend' - `list` EvalAssertion[] - union — An SDK assertion reported through UVC. Discovery assertions require complete raw tool declarations; missing evidence is an evaluator error. Observation kinds, including noDeprecatedToolExposed, require role advisory. Structural schema; runtime also validates policy and unique required annotation keys. - object - `type` 'toolDescriptionsPresent', required - `minLength` integer - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'toolAnnotationsPresent', required - `require` string[] - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'toolNamesUnique', required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'noDeprecatedToolExposed', required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'toolInputSchemasWellFormed', required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'toolOutputSchemasPresent', required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'toolCalledWith', required - `toolName` string, required - `args` object, required - … - `minCount` integer - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'toolCalledAtLeastOnce', required - `toolName` string, required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'toolNeverCalled', required - `toolName` string, required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'onlyToolsCalled', required - `toolNames` string[], required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'firstToolWas', required - `toolName` string, required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'responseContains', required - `needle` string, required - `caseSensitive` boolean - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'responseMatches', required - `pattern` string, required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'noToolErrors', required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'finalAssistantMessageNonEmpty', required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'tokenBudgetUnder', required - `tokens` integer, required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'widgetRendered', required - `toolName` string - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'widgetRenderLatencyUnder', required - `ms` integer, required - `toolName` string - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'widgetNoConsoleErrors', required - `toolName` string - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'turnCountUnder', required - `turns` integer, required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'noEndingQuestion', required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'toolLatencyUnder', required - `ms` integer, required - `toolName` string - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'toolResultContains', required - `needle` string, required - `caseSensitive` boolean - `toolName` string - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'toolResultMatchesSchema', required - `schema` unknown, required - `toolName` string - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'toolResultSizeUnder', required - `maxBytes` integer, required - `toolName` string - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'argumentsMatchToolSchema', required - `toolName` string - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'noRepeatedIdenticalCall', required - `toolName` string - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'toolCallCountUnder', required - `count` integer, required - `toolName` string - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'toolCalledBefore', required - `toolName` string, required - `beforeToolName` string, required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'noDeprecatedToolCalled', required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'noDestructiveToolCalled', required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'toolErrorNamesInput', required - `toolName` string - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'fullPageHasContinuation', required - `toolName` string - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - `import` EvalCaseImportClaim — What a converter CLAIMED about one imported case. `exact` is CONVERTER-CLAIMED exact — the converter says it applied a structural mapping rule, cited in `note`. MCPJam has NOT verified semantic equivalence, so user-facing copy must say "claimed exact", never "verified" or "accepted". Claim-only: who approved an approximation, when, and why is a PER-RUN decision frozen on the run (`ImportEligibility.approvedApproximationReceipts`), never stored on the case — an approval that lived on a case would outlive the run it was granted for and the edit that invalidated it. Approval and internal keys are rejected with 400, never stripped. - `status` 'exact' | 'approximated' | 'unsupported' | 'unresolved', required — `exact`: the converter claims a 1:1 structural mapping, and must cite it in `note`. `approximated`: behaviour was intentionally approximated; a human must approve it for EVERY run. `unsupported`: the source behaviour cannot currently be represented. `unresolved`: a deterministic reference does not resolve against the live target. A selected `unsupported` or `unresolved` case cannot run. - `sourceCaseKey` string — The case's identity in the source system, when it had one. - `note` string — Why the status is what it is — the mapping rule cited, or what was lost. REQUIRED when `status` is `exact`. - `source` EvalCaseSource — Where an AI-assisted Markdown case was authored from. Provenance only: the file, its hash, and the line range the extractor read, so a reader can trace a case back to its source document. It is not an import claim (see `EvalCaseImportClaim`) and carries no approval semantics. - `format` 'markdown', required - `method` 'ai', required — How the case was derived from the document. `ai`: an extractor model proposed it and a person reviewed it before it was saved. - `fileName` string, required - `fileHash` string, required — SHA-256 of the source document, lowercase hex. - `excerpt` string, required — The passage the case was extracted from. - `startLine` integer, required - `endLine` integer, required — Inclusive; never less than `startLine`. - `extractorVersion` string, required - `createdAt` number, nullable - `updatedAt` number, nullable - `suppressedSuiteStandardCheckIds` string[] — Stable standard assertion family IDs suppressed from suite defaults. Applies before inherit/extend/replace resolution; explicit case and step assertions are preserved. Omitted updates preserve; [] clears. At most 64 IDs, normalized to unique sorted values. - `counts` object, required - `normal` integer - `negative` integer - `skipped` object[] — Drafts that were generated but failed to persist. Surfaced rather than silently dropped. ## Other responses - `400` — Malformed body or parameters. - `401` — Missing, invalid, revoked, or orphaned key (`UNAUTHORIZED`) — or the **target MCP server** needs an OAuth grant (`OAUTH_REQUIRED`), which is a property of the server, not your key. - `403` — Key is valid but not allowed to do this. - `404` — Unknown project, server, or resource. - `409` — The resource is not in a state that accepts this write — a stale `expectedRevision`, a duplicate name, or an environment that cannot currently be launched. The request was well-formed; re-read the resource and retry. - `429` — Per-key rate limit exceeded (60 requests/minute sustained, bursts up to 10). Honor `Retry-After` and back off with jitter. - `500` — Something failed on MCPJam's side. - `502` — Could not connect to the target MCP server. - `504` — The target MCP server connected but didn't respond in time. ## Changes - **2026-09-16** `0e05c5e4277f` — 1 info - added the optional property `created/items/judge` to the response with the `200` status - **2026-09-14** `56b9d1dda7ea` — 1 breaking, 2 info - added `subschema #1, subschema #2, subschema #3, subschema #4, subschema #5, subschema #6, subschema #7, subschema #8, subschema #9, subschema #10, subschema #11, subschema #12, subschema #13, subschema #14, subschema #15, subschema #16, subschema #17, subschema #18, subschema #19, subschema #20, subschema #21, subschema #22, subschema #23, subschema #24, subschema #25, subschema #26, subschema #27, subschema #28, subschema #29, subschema #30, subschema #31, subschema #32, subschema #33` to the `created/items/checks/list/items/` response property `oneOf` list for the response status `200` - added the new optional `header` request parameter `x-mcpjam-eval-vocabulary` - removed `subschema #1, subschema #2, subschema #3, subschema #4, subschema #5, subschema #6, subschema #7, subschema #8, subschema #9, subschema #10, subschema #11, subschema #12, subschema #13, subschema #14, subschema #15, subschema #16, subschema #17, subschema #18, subschema #19, subschema #20, subschema #21, subschema #22, subschema #23, subschema #24, subschema #25, subschema #26, subschema #27, subschema #28, subschema #29, subschema #30, subschema #31, subschema #32, subschema #33` from the `created/items/checks/list/items/` response property `oneOf` list for the response status `200` - …earlier changes not shown [Full history](https://skmtc.dev/mcpjam/apis/mcpjam-api/changes/projects/:projectId/eval-suites/:suiteId/cases/generate/post.md) --- [API](https://skmtc.dev/mcpjam/apis/mcpjam-api.md) · [All operations](https://skmtc.dev/mcpjam/apis/mcpjam-api/llms.txt) · [OpenAPI document](https://skmtc.dev/mcpjam/apis/mcpjam-api/revisions/0e05c5e4277f?raw)