--- title: "Author eval cases from a document" method: POST path: "/projects/{projectId}/eval-suites/{suiteId}/cases/import" tags: ["Eval runs"] --- # Author eval cases from a document `POST /projects/{projectId}/eval-suites/{suiteId}/cases/import` The sibling of `cases/generate`: generation invents cases from the suite's tools, import reads them out of something a person already wrote — a test plan, a spreadsheet of scenarios, a JSON export. MCPJam's own model turns the document into full cases (ordered steps, tool calls, assertions), so the caller does not have to guess the case shape. No format is declared: the model reads the document's shape itself, so Markdown, JSON, CSV and plain notes all work. **Spends the organization's credits**, unlike generation, which is on MCPJam. Pass an idempotency key so a retry replays the same job instead of authoring — and billing — the document twice. Like `cases/generate`, this discovers the suite's tools over a live MCP connection first, so the authored cases are grounded in tools the suite can actually call. Answers `200` when the job finishes inside the wait window and `202` with a `jobId` when it does not — poll the job, then commit it. A case the commit could not write comes back under `skipped`, with a `reviewUrl` that opens the app on exactly those drafts. ## Path parameters - `projectId` string, required - `suiteId` string, required ## Headers - `x-mcpjam-idempotency-key` string ## Request body - EvalCaseImportRequest - `content` string, required — The document, as UTF-8 text. Markdown, JSON, CSV or plain notes — no format is declared, because the authoring model reads the document's shape itself. At most 100 KiB. - `fileName` string — Recorded on each case's provenance so a reviewer can find the source. Optional: a pasted document has no file behind it. - `servers` string[] — Server ids or names to discover tools from. Mutually exclusive with `environmentId`. - `environmentId` string — Author against this environment's closed server set. Mutually exclusive with `servers`. - `caseModels` object[] - `model` string - `provider` string - `duplicatePolicy` 'block' | 'warn' | 'create_anyway' — Applied when the cases are written, not when the job starts. Anything but `block` requires `overrideReason`. - `overrideReason` string - `idempotencyKey` string — Makes a retry replay the recorded job instead of authoring — and billing — the document a second time. ## Response `200` The cases the job authored and committed, or the job's own status when it finished without committing (failed, cancelled, or still pending at the end of the wait). - union - EvalCaseImported - `generationModel` string, required - `created` EvalCase[], required - `judge` object — Per-case judge override. enabled: false skips goal-completion grading for this case. - `enabled` boolean - `id` string, required - `declaredId` string — The case's effective declared id. Absent on cases authored before declared identity existed. - `title` string, required - `steps` EvalTestStep[], required — Ordered test steps. A `prompt` step is a model turn; a single model-free `toolCall` step is a render-check; `assert` steps hold the expectations. - `id` string, required - `kind` 'prompt' | 'toolCall' | 'interact' | 'assert', required - `prompt` string — User message (`kind: prompt`). - `serverName` string — Server that owns the tool (`kind: toolCall`). - `toolName` string — Tool name (`kind: toolCall` / `interact`). - `arguments` object — Tool-call arguments (`kind: toolCall`). - `action` object — Widget action (`kind: interact`). - `assertion` object — The rule an `assert` step checks (`kind: assert`). NOT a narrower spelling of a case's `checks`: this is `WidgetAssertion | Predicate`, and that union is why the field is called an assertion rather than a check. A `Predicate` is evaluated against the PERSISTED transcript, so a stored run can be re-graded against it months later; a `WidgetAssertion` is evaluated against a LIVE DOM and can never be replayed. Calling this a check would promise the replayability only one half of it has. - `expectedOutput` string - `iterations` integer, required - `repetitions` integer — Trials this case runs under per-case grading, overriding the suite default. Absent means the case inherits it. NOT a second spelling of `iterations`: that one is read as a FLOOR (`max(iterations, suite.minimumIterations)`) by a suite decided on suite-wide accuracy, and a per-case-graded case still reports it for compatibility. This one REPLACES the count rather than raising it — a case at 7 runs 7 times under a floor of 3 and 3 times under a default of 3. - `passThreshold` number — Fraction of this case's trials that must pass, overriding the suite default. Absent means the case inherits it. Never derived from the suite's `minimumAccuracy`, which is a PERCENT under a different resolver. - `isNegative` boolean, required — When true, the case passes if NO tools are called. - `scenario` string - `intent` string — Optional authored analytics grouping label. Must be already trimmed; absent means unlabelled. - `kind` 'capability' | 'regression' — Authored case kind for the simple editor. Absent means the editor derives it from matchOptions. - `models` object[], required - `model` string, required - `provider` string - `matchOptions` object — Absent when the case sets none — omitted from the response rather than sent as `null`. - `checks` object — Absent when the case sets none — omitted from the response rather than sent as `null`. - `mode` 'inherit' | 'replace' | 'extend' - `list` EvalAssertion[] - union — An SDK assertion reported through UVC. Discovery assertions require complete raw tool declarations; missing evidence is an evaluator error. Observation kinds, including noDeprecatedToolExposed, require role advisory. Structural schema; runtime also validates policy and unique required annotation keys. - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - `import` EvalCaseImportClaim — What a converter CLAIMED about one imported case. `exact` is CONVERTER-CLAIMED exact — the converter says it applied a structural mapping rule, cited in `note`. MCPJam has NOT verified semantic equivalence, so user-facing copy must say "claimed exact", never "verified" or "accepted". Claim-only: who approved an approximation, when, and why is a PER-RUN decision frozen on the run (`ImportEligibility.approvedApproximationReceipts`), never stored on the case — an approval that lived on a case would outlive the run it was granted for and the edit that invalidated it. Approval and internal keys are rejected with 400, never stripped. - `status` 'exact' | 'approximated' | 'unsupported' | 'unresolved', required — `exact`: the converter claims a 1:1 structural mapping, and must cite it in `note`. `approximated`: behaviour was intentionally approximated; a human must approve it for EVERY run. `unsupported`: the source behaviour cannot currently be represented. `unresolved`: a deterministic reference does not resolve against the live target. A selected `unsupported` or `unresolved` case cannot run. - `sourceCaseKey` string — The case's identity in the source system, when it had one. - `note` string — Why the status is what it is — the mapping rule cited, or what was lost. REQUIRED when `status` is `exact`. - `source` EvalCaseSource — Where an AI-assisted Markdown case was authored from. Provenance only: the file, its hash, and the line range the extractor read, so a reader can trace a case back to its source document. It is not an import claim (see `EvalCaseImportClaim`) and carries no approval semantics. - `format` 'markdown', required - `method` 'ai', required — How the case was derived from the document. `ai`: an extractor model proposed it and a person reviewed it before it was saved. - `fileName` string, required - `fileHash` string, required — SHA-256 of the source document, lowercase hex. - `excerpt` string, required — The passage the case was extracted from. - `startLine` integer, required - `endLine` integer, required — Inclusive; never less than `startLine`. - `extractorVersion` string, required - `createdAt` number, nullable - `updatedAt` number, nullable - `suppressedSuiteStandardCheckIds` string[] — Stable standard assertion family IDs suppressed from suite defaults. Applies before inherit/extend/replace resolution; explicit case and step assertions are preserved. Omitted updates preserve; [] clears. At most 64 IDs, normalized to unique sorted values. - `counts` object, required - `normal` integer - `negative` integer - `skipped` object[] — Cases the commit could not write, each with its reason. They stay on the job rather than being lost. - `reviewUrl` string, uri — Opens the app on exactly the skipped drafts. Present only when something was skipped — finishing them there costs nothing, whereas re-sending the document re-authors and re-bills every case in it. - EvalAuthoringJobAccepted - `jobId` string, required — Poll this job, then commit it. - `status` 'pending' | 'completed' | 'failed' | 'cancelled', required - `error` string — Why the job stopped. Present only on a failed job. - `generationModel` string - `created` EvalCase[] — Always empty here — nothing is written until the job is committed. - `judge` object — Per-case judge override. enabled: false skips goal-completion grading for this case. - `enabled` boolean - `id` string, required - `declaredId` string — The case's effective declared id. Absent on cases authored before declared identity existed. - `title` string, required - `steps` EvalTestStep[], required — Ordered test steps. A `prompt` step is a model turn; a single model-free `toolCall` step is a render-check; `assert` steps hold the expectations. - `id` string, required - `kind` 'prompt' | 'toolCall' | 'interact' | 'assert', required - `prompt` string — User message (`kind: prompt`). - `serverName` string — Server that owns the tool (`kind: toolCall`). - `toolName` string — Tool name (`kind: toolCall` / `interact`). - `arguments` object — Tool-call arguments (`kind: toolCall`). - `action` object — Widget action (`kind: interact`). - `assertion` object — The rule an `assert` step checks (`kind: assert`). NOT a narrower spelling of a case's `checks`: this is `WidgetAssertion | Predicate`, and that union is why the field is called an assertion rather than a check. A `Predicate` is evaluated against the PERSISTED transcript, so a stored run can be re-graded against it months later; a `WidgetAssertion` is evaluated against a LIVE DOM and can never be replayed. Calling this a check would promise the replayability only one half of it has. - `expectedOutput` string - `iterations` integer, required - `repetitions` integer — Trials this case runs under per-case grading, overriding the suite default. Absent means the case inherits it. NOT a second spelling of `iterations`: that one is read as a FLOOR (`max(iterations, suite.minimumIterations)`) by a suite decided on suite-wide accuracy, and a per-case-graded case still reports it for compatibility. This one REPLACES the count rather than raising it — a case at 7 runs 7 times under a floor of 3 and 3 times under a default of 3. - `passThreshold` number — Fraction of this case's trials that must pass, overriding the suite default. Absent means the case inherits it. Never derived from the suite's `minimumAccuracy`, which is a PERCENT under a different resolver. - `isNegative` boolean, required — When true, the case passes if NO tools are called. - `scenario` string - `intent` string — Optional authored analytics grouping label. Must be already trimmed; absent means unlabelled. - `kind` 'capability' | 'regression' — Authored case kind for the simple editor. Absent means the editor derives it from matchOptions. - `models` object[], required - `model` string, required - `provider` string - `matchOptions` object — Absent when the case sets none — omitted from the response rather than sent as `null`. - `checks` object — Absent when the case sets none — omitted from the response rather than sent as `null`. - `mode` 'inherit' | 'replace' | 'extend' - `list` EvalAssertion[] - union — An SDK assertion reported through UVC. Discovery assertions require complete raw tool declarations; missing evidence is an evaluator error. Observation kinds, including noDeprecatedToolExposed, require role advisory. Structural schema; runtime also validates policy and unique required annotation keys. - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - object - … - `import` EvalCaseImportClaim — What a converter CLAIMED about one imported case. `exact` is CONVERTER-CLAIMED exact — the converter says it applied a structural mapping rule, cited in `note`. MCPJam has NOT verified semantic equivalence, so user-facing copy must say "claimed exact", never "verified" or "accepted". Claim-only: who approved an approximation, when, and why is a PER-RUN decision frozen on the run (`ImportEligibility.approvedApproximationReceipts`), never stored on the case — an approval that lived on a case would outlive the run it was granted for and the edit that invalidated it. Approval and internal keys are rejected with 400, never stripped. - `status` 'exact' | 'approximated' | 'unsupported' | 'unresolved', required — `exact`: the converter claims a 1:1 structural mapping, and must cite it in `note`. `approximated`: behaviour was intentionally approximated; a human must approve it for EVERY run. `unsupported`: the source behaviour cannot currently be represented. `unresolved`: a deterministic reference does not resolve against the live target. A selected `unsupported` or `unresolved` case cannot run. - `sourceCaseKey` string — The case's identity in the source system, when it had one. - `note` string — Why the status is what it is — the mapping rule cited, or what was lost. REQUIRED when `status` is `exact`. - `source` EvalCaseSource — Where an AI-assisted Markdown case was authored from. Provenance only: the file, its hash, and the line range the extractor read, so a reader can trace a case back to its source document. It is not an import claim (see `EvalCaseImportClaim`) and carries no approval semantics. - `format` 'markdown', required - `method` 'ai', required — How the case was derived from the document. `ai`: an extractor model proposed it and a person reviewed it before it was saved. - `fileName` string, required - `fileHash` string, required — SHA-256 of the source document, lowercase hex. - `excerpt` string, required — The passage the case was extracted from. - `startLine` integer, required - `endLine` integer, required — Inclusive; never less than `startLine`. - `extractorVersion` string, required - `createdAt` number, nullable - `updatedAt` number, nullable - `suppressedSuiteStandardCheckIds` string[] — Stable standard assertion family IDs suppressed from suite defaults. Applies before inherit/extend/replace resolution; explicit case and step assertions are preserved. Omitted updates preserve; [] clears. At most 64 IDs, normalized to unique sorted values. - `counts` object - `normal` integer - `negative` integer ## Other responses - `202` — The authoring job is still running when the wait window closes. Nothing has been written to the suite yet: poll the job, then commit it. - `400` — Malformed body or parameters. - `401` — Missing, invalid, revoked, or orphaned key (`UNAUTHORIZED`) — or the **target MCP server** needs an OAuth grant (`OAUTH_REQUIRED`), which is a property of the server, not your key. - `403` — Key is valid but not allowed to do this. - `404` — Unknown project, server, or resource. - `409` — The resource is not in a state that accepts this write — a stale `expectedRevision`, a duplicate name, or an environment that cannot currently be launched. The request was well-formed; re-read the resource and retry. - `429` — Per-key rate limit exceeded (60 requests/minute sustained, bursts up to 10). Honor `Retry-After` and back off with jitter. - `500` — Something failed on MCPJam's side. - `502` — Could not connect to the target MCP server. - `504` — The target MCP server connected but didn't respond in time. ## Changes - **2026-09-25** `311db0744979` — 1 info - endpoint added [Change history](https://skmtc.dev/mcpjam/apis/mcpjam-api/changes/projects/:projectId/eval-suites/:suiteId/cases/import/post.md) --- [API](https://skmtc.dev/mcpjam/apis/mcpjam-api.md) · [All operations](https://skmtc.dev/mcpjam/apis/mcpjam-api/llms.txt) · [OpenAPI document](https://skmtc.dev/mcpjam/apis/mcpjam-api/revisions/a15a9864fb85?raw)