--- title: "Create several eval cases" method: POST path: "/projects/{projectId}/eval-suites/{suiteId}/cases/batch" tags: ["Eval runs"] --- # Create several eval cases `POST /projects/{projectId}/eval-suites/{suiteId}/cases/batch` Adds up to 100 cases to the suite in one call — the bulk form of the single create, and the way to import a suite or convert a repo's test files without a round trip per case. Every entry needs a `title` and a non-empty `steps` array. Cases are validated together and reported individually: a `failed` entry does not roll back its committed siblings, so the response is `201` even when some entries were refused — read `failed` rather than branching on the status alone. Send an `Idempotency-Key` header to make a retry land on the same rows. ## Path parameters - `projectId` string, required - `suiteId` string, required ## Query parameters - `declaredSuiteId` string ## Headers - `x-mcpjam-eval-vocabulary` '1' | '2' - `Idempotency-Key` string ## Request body - EvalCasesBatchCreateRequest — Author several cases in one call. Each entry takes the same fields as a single create. The 100-case cap is deliberately smaller than a suite file's 500-case limit, so a maximal file uploads in several calls. - `declaredSuiteId` string — The suite file's own `suite.id`, when THIS request is a suite-file sync. A CI-owned suite (see `EvalSuiteDetail.managedBy`) refuses configuration writes with `409` and `details.reason: "CI_OWNED_SUITE_READ_ONLY"`. Naming the suite's own declared id is how the file writing itself is allowed through — the platform permits the write only when the id matches, so naming any other id refuses exactly as loudly as naming none. Omit it for ordinary edits. PREFER THE `declaredSuiteId` QUERY PARAMETER on this route. This body field still works here, but these request bodies are strict on every Inspector that predates the CI-owned lock, so a body field is a `400` against an older deployment while an unknown query parameter is simply ignored. The MCPJam SDK and CLI send the query parameter. - `cases` EvalCaseCreateRequest[], required - `judge` object, nullable — Per-case judge override. enabled: false skips goal-completion grading for this case; null clears the override and restores the suite setting. - `enabled` boolean - `declaredSuiteId` string — The suite file's own `suite.id`, when THIS request is a suite-file sync. A CI-owned suite (see `EvalSuiteDetail.managedBy`) refuses configuration writes with `409` and `details.reason: "CI_OWNED_SUITE_READ_ONLY"`. Naming the suite's own declared id is how the file writing itself is allowed through — the platform permits the write only when the id matches, so naming any other id refuses exactly as loudly as naming none. Omit it for ordinary edits. PREFER THE `declaredSuiteId` QUERY PARAMETER on this route. This body field still works here, but these request bodies are strict on every Inspector that predates the CI-owned lock, so a body field is a `400` against an older deployment while an unknown query parameter is simply ignored. The MCPJam SDK and CLI send the query parameter. - `title` string, required - `id` string — Stable declared id for the case — the id it answers to in a suite file, an import or a CLI argument. Minted for you when omitted, and unique within the suite. Distinct from the `id` on the response, which is the case's platform row id. - `steps` EvalTestStep[], required — REPLACES the case's test definition wholesale when provided. - `id` string, required - `kind` 'prompt' | 'toolCall' | 'interact' | 'assert', required - `prompt` string — User message (`kind: prompt`). - `serverName` string — Server that owns the tool (`kind: toolCall`). - `toolName` string — Tool name (`kind: toolCall` / `interact`). - `arguments` object — Tool-call arguments (`kind: toolCall`). - `action` object — Widget action (`kind: interact`). - `assertion` object — The rule an `assert` step checks (`kind: assert`). NOT a narrower spelling of a case's `checks`: this is `WidgetAssertion | Predicate`, and that union is why the field is called an assertion rather than a check. A `Predicate` is evaluated against the PERSISTED transcript, so a stored run can be re-graded against it months later; a `WidgetAssertion` is evaluated against a LIVE DOM and can never be replayed. Calling this a check would promise the replayability only one half of it has. - `expectedOutput` string - `iterations` integer - `repetitions` integer — Per-case repetitions; omitted cases inherit the suite default. PER-CASE GRADING ONLY — on a suite decided by the suite-wide accuracy threshold this is a VALIDATION_ERROR naming the criterion change that would make it readable, because that criterion takes the trial count from `iterations` raised to the suite's `minimumIterations` and would never read this. Capped at 10, the same hosted ceiling as `iterations`; the suite-file contract's larger `MAX_REPETITIONS` is what a file may DECLARE, not what a hosted run accepts. - `passThreshold` number — Per-case pass threshold FRACTION: the share of this case's own iterations that must pass. PER-CASE GRADING ONLY — on a suite decided by the suite-wide accuracy threshold this is a VALIDATION_ERROR, for the same reason as the count: that criterion has one percentage over the whole run and no per-case threshold to override. - `isNegative` boolean - `scenario` string - `intent` string — Optional authored analytics grouping label. Must be already trimmed. - `kind` 'capability' | 'regression' — Authored case kind for the simple editor. Absent means the editor derives it from matchOptions. - `models` object[] - `model` string, required - `provider` string — Derived from a `provider/model` id when omitted. - `matchOptions` object, nullable — `null` clears the case override and inherits the suite's. - `checks` object, nullable — `null` clears the case override. - `mode` 'inherit' | 'replace' | 'extend' - `list` EvalAssertion[] - union — An SDK assertion reported through UVC. Discovery assertions require complete raw tool declarations; missing evidence is an evaluator error. Observation kinds, including noDeprecatedToolExposed, require role advisory. Structural schema; runtime also validates policy and unique required annotation keys. - object - `type` 'toolDescriptionsPresent', required - `minLength` integer - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'toolAnnotationsPresent', required - `require` string[] - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'toolNamesUnique', required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'noDeprecatedToolExposed', required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'toolInputSchemasWellFormed', required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'toolOutputSchemasPresent', required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'toolCalledWith', required - `toolName` string, required - `args` object, required - … - `minCount` integer - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'toolCalledAtLeastOnce', required - `toolName` string, required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'toolNeverCalled', required - `toolName` string, required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'onlyToolsCalled', required - `toolNames` string[], required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'firstToolWas', required - `toolName` string, required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'responseContains', required - `needle` string, required - `caseSensitive` boolean - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'responseMatches', required - `pattern` string, required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'noToolErrors', required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'finalAssistantMessageNonEmpty', required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'tokenBudgetUnder', required - `tokens` integer, required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'widgetRendered', required - `toolName` string - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'widgetRenderLatencyUnder', required - `ms` integer, required - `toolName` string - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'widgetNoConsoleErrors', required - `toolName` string - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'turnCountUnder', required - `turns` integer, required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'noEndingQuestion', required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'toolLatencyUnder', required - `ms` integer, required - `toolName` string - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'toolResultContains', required - `needle` string, required - `caseSensitive` boolean - `toolName` string - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'toolResultMatchesSchema', required - `schema` unknown, required - `toolName` string - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'toolResultSizeUnder', required - `maxBytes` integer, required - `toolName` string - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'argumentsMatchToolSchema', required - `toolName` string - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'noRepeatedIdenticalCall', required - `toolName` string - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'toolCallCountUnder', required - `count` integer, required - `toolName` string - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'toolCalledBefore', required - `toolName` string, required - `beforeToolName` string, required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'noDeprecatedToolCalled', required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'noDestructiveToolCalled', required - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'toolErrorNamesInput', required - `toolName` string - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - object - `type` 'fullPageHasContinuation', required - `toolName` string - `role` 'gating' | 'advisory' | 'required' - `severity` 'warn' - `import` EvalCaseImportClaim — What a converter CLAIMED about one imported case. `exact` is CONVERTER-CLAIMED exact — the converter says it applied a structural mapping rule, cited in `note`. MCPJam has NOT verified semantic equivalence, so user-facing copy must say "claimed exact", never "verified" or "accepted". Claim-only: who approved an approximation, when, and why is a PER-RUN decision frozen on the run (`ImportEligibility.approvedApproximationReceipts`), never stored on the case — an approval that lived on a case would outlive the run it was granted for and the edit that invalidated it. Approval and internal keys are rejected with 400, never stripped. - `status` 'exact' | 'approximated' | 'unsupported' | 'unresolved', required — `exact`: the converter claims a 1:1 structural mapping, and must cite it in `note`. `approximated`: behaviour was intentionally approximated; a human must approve it for EVERY run. `unsupported`: the source behaviour cannot currently be represented. `unresolved`: a deterministic reference does not resolve against the live target. A selected `unsupported` or `unresolved` case cannot run. - `sourceCaseKey` string — The case's identity in the source system, when it had one. - `note` string — Why the status is what it is — the mapping rule cited, or what was lost. REQUIRED when `status` is `exact`. - `source` EvalCaseSource — Where an AI-assisted Markdown case was authored from. Provenance only: the file, its hash, and the line range the extractor read, so a reader can trace a case back to its source document. It is not an import claim (see `EvalCaseImportClaim`) and carries no approval semantics. - `format` 'markdown', required - `method` 'ai', required — How the case was derived from the document. `ai`: an extractor model proposed it and a person reviewed it before it was saved. - `fileName` string, required - `fileHash` string, required — SHA-256 of the source document, lowercase hex. - `excerpt` string, required — The passage the case was extracted from. - `startLine` integer, required - `endLine` integer, required — Inclusive; never less than `startLine`. - `extractorVersion` string, required - `suppressedSuiteStandardCheckIds` string[] — Stable standard assertion family IDs suppressed from suite defaults. Applies before inherit/extend/replace resolution; explicit case and step assertions are preserved. Omitted updates preserve; [] clears. At most 64 IDs, normalized to unique sorted values. - `duplicatePolicy` 'block' | 'warn' | 'create_anyway' — What to do with a case whose definition already matches one in the suite. An unrecognized value coerces to `block` and reports the coercion in the response's `duplicatePolicy` rather than failing the call. - `overrideReason` string — Required by `warn` and `create_anyway`. Recorded on the case's revision. ## Response `201` Per-case outcomes. Some entries may have failed; see `failed`. - EvalCasesBatchResult — A partial outcome by design: the cases in `created` were written even when `failed` is non-empty. Both arrays carry the `index` of the request entry they describe. - `created` object[], required - `index` integer, required — Position in the request's `cases` array. - `id` string, required — Platform case id — the path parameter for the per-case routes. - `declaredId` string — The effective declared id. On a replay this is the stored case's, not the one just sent. - `title` string, required - `replayed` boolean, required — True when an idempotent retry landed on an already-authored case; nothing new was written. - `warnings` EvalCaseWarning[] - `code` string, required - `message` string, required - `failed` object[], required - `index` integer, required - `title` string - `declaredId` string - `code` 'INVALID_ITEM' | 'INVALID_CASE_ID' | 'DUPLICATE_CASE_ID' | 'DUPLICATE_IDEMPOTENCY_KEY' | 'IDEMPOTENCY_CONFLICT' | 'DUPLICATE_CONTENT' | 'OVERRIDE_REASON_REQUIRED' | 'INVALID_CASE' | 'INVALID_IMPORT' | 'UNSTORABLE_ITEM' | 'ITEM_TOO_LARGE', required — Stable machine-readable reason. - `message` string, required - `duplicatePolicy` object, required — What policy actually applied, so a coercion is never silent. - `requestedPolicy` string - `effectivePolicy` string, required - `coerced` boolean, required - `warnings` EvalCaseWarning[] - `code` string, required - `message` string, required ## Other responses - `400` — Malformed body or parameters. - `401` — Missing, invalid, revoked, or orphaned key (`UNAUTHORIZED`) — or the **target MCP server** needs an OAuth grant (`OAUTH_REQUIRED`), which is a property of the server, not your key. - `403` — Key is valid but not allowed to do this. - `404` — Unknown project, server, or resource. - `429` — Per-key rate limit exceeded (60 requests/minute sustained, bursts up to 10). Honor `Retry-After` and back off with jitter. - `500` — Something failed on MCPJam's side. - `502` — Could not connect to the target MCP server. ## Changes - **2026-09-16** `0e05c5e4277f` — 1 info - added the new optional request property `cases/items/judge` - **2026-09-14** `56b9d1dda7ea` — 1 breaking, 2 info - removed `subschema #1, subschema #2, subschema #3, subschema #4, subschema #5, subschema #6, subschema #7, subschema #8, subschema #9, subschema #10, subschema #11, subschema #12, subschema #13, subschema #14, subschema #15, subschema #16, subschema #17, subschema #18, subschema #19, subschema #20, subschema #21, subschema #22, subschema #23, subschema #24, subschema #25, subschema #26, subschema #27, subschema #28, subschema #29, subschema #30, subschema #31, subschema #32, subschema #33` from the `cases/items/checks/list/items/` request property `oneOf` list - added the new optional `header` request parameter `x-mcpjam-eval-vocabulary` - added `subschema #1, subschema #2, subschema #3, subschema #4, subschema #5, subschema #6, subschema #7, subschema #8, subschema #9, subschema #10, subschema #11, subschema #12, subschema #13, subschema #14, subschema #15, subschema #16, subschema #17, subschema #18, subschema #19, subschema #20, subschema #21, subschema #22, subschema #23, subschema #24, subschema #25, subschema #26, subschema #27, subschema #28, subschema #29, subschema #30, subschema #31, subschema #32, subschema #33` to the `cases/items/checks/list/items/` request property `oneOf` list - …earlier changes not shown [Full history](https://skmtc.dev/mcpjam/apis/mcpjam-api/changes/projects/:projectId/eval-suites/:suiteId/cases/batch/post.md) --- [API](https://skmtc.dev/mcpjam/apis/mcpjam-api.md) · [All operations](https://skmtc.dev/mcpjam/apis/mcpjam-api/llms.txt) · [OpenAPI document](https://skmtc.dev/mcpjam/apis/mcpjam-api/revisions/0e05c5e4277f?raw)