--- title: "Update an eval suite's settings" method: PATCH path: "/projects/{projectId}/eval-suites/{suiteId}" tags: ["Eval runs"] --- # Update an eval suite's settings `PATCH /projects/{projectId}/eval-suites/{suiteId}` Partial update — omitted fields are left as they are. `environmentIds` is tri-state: a non-empty array sets/replaces, `null` clears (reverting the suite to legacy config), and `[]` is rejected rather than silently treated as a clear. Responds with the full updated suite. ## Path parameters - `projectId` string, required - `suiteId` string, required ## Query parameters - `declaredSuiteId` string ## Headers - `x-mcpjam-eval-vocabulary` '1' | '2' ## Request body - EvalSuiteUpdateRequest — Partial update of a suite's settings. Omitted fields are left as they are. - `declaredSuiteId` string — The suite file's own `suite.id`, when THIS request is a suite-file sync. A CI-owned suite (see `EvalSuiteDetail.managedBy`) refuses configuration writes with `409` and `details.reason: "CI_OWNED_SUITE_READ_ONLY"`. Naming the suite's own declared id is how the file writing itself is allowed through — the platform permits the write only when the id matches, so naming any other id refuses exactly as loudly as naming none. Omit it for ordinary edits. PREFER THE `declaredSuiteId` QUERY PARAMETER on this route. This body field still works here, but these request bodies are strict on every Inspector that predates the CI-owned lock, so a body field is a `400` against an older deployment while an unknown query parameter is simply ignored. The MCPJam SDK and CLI send the query parameter. - `name` string - `description` string - `environment` object — LEGACY server bag. Unrelated to `environmentIds`. - `servers` string[] - `computerEnvironment` string, nullable — Sandbox-image name or id. `null` clears the pin (runs fall back to the provider's default base image). Enumerate the choices with `list_sandbox_images`. - `environmentIds` string[], nullable — Tri-state: a non-empty array sets/replaces the attachments, `null` CLEARS them (reverting the suite to legacy config), and `[]` is REJECTED rather than silently read as a clear. - `executionConfig` object - `model` string - `systemPrompt` string - `temperature` number - `hosts` object[] - `host` string, required — Host name or id. - `servers` string[] — Server names (as bound in the suite's environment) or project-server ids. - `settings` object - `minimumAccuracy` number — Minimum pass rate as a PERCENT, 0-100. Legacy suites only; v2 suites use `passThreshold` (a fraction) and refuse this field. - `minimumIterations` integer, nullable — Suite-level floor on per-case iterations: every case runs at least this many times (`max(case.iterations, minimumIterations)`). `null` clears it. - `matchOptions` object, nullable - `checks` object[], nullable - `judge` object - `enabled` boolean - `model` string - `autoRun` boolean — The flag the grader actually gates on. Without it a suite can be `enabled` forever and never grade a run. - `threshold` number - `severity` 'warn' — Presentation severity on the goal-completion slot. Legal only with an advisory role. - `rubric` object, nullable — The suite's own grading criteria, handed to the judge alongside each case's expected output. `null` CLEARS them; an empty `criteria` array is refused, because a rubric that asks nothing still changes what the judge was asked. Editing this retires the suite's judge calibration. - `criteria` object[], required - `id` string, required — Stable id the judge cites in its reasons. Unique within the rubric. - `label` string, required - `description` string - `required` boolean - `repetitions` integer — PER-CASE GRADING only: the DEFAULT trials per case, which a case's own count REPLACES (unlike `minimumIterations`, a floor that raises it). On a suite decided by the suite-wide accuracy threshold, sending this together with `passThreshold` CHANGES which criterion decides the suite; neither alone is accepted there, because neither half answers what a case is graded against. That change re-decides every case in the suite, which is why it takes both fields rather than happening as a side effect of one. - `passThreshold` number — PER-CASE GRADING only: the FRACTION of a case's own trials that must pass, 0-1 (0.8 is eighty percent). A DIFFERENT criterion from `minimumAccuracy`, not a unit conversion of it — that one is a single percentage over the whole run — so sending both is refused rather than reconciled. - `validity` object — PER-CASE GRADING only: when a run's measurement is trustworthy enough to decide, so a run that measured too little is `inconclusive` rather than passed or failed. A suite decided by the suite-wide accuracy threshold has no validity phase at all and accepts nothing here. Omitted members keep the contract defaults (`minCompletionRate` 0.8, `maxEvaluatorErrorRate` 0.1, and an omitted `minEligibleTrials` requiring every configured trial attempted plus at least one gradeable — the STRICTER rule, not no minimum); supplied members merge over the suite's stored validity rather than replacing it. - `minEligibleTrials` integer - `minCompletionRate` number - `maxEvaluatorErrorRate` number - `expectedRevisionNumber` integer — The suite's `revisionNumber` as you last read it. Supplying it makes this edit a compare-and-set: a suite changed since then is refused with 409 having written nothing. Omit for last-write-wins. ## Response `200` The updated suite. - EvalSuiteDetail — One eval suite's full configuration. Distinct from the `EvalSuite` summary returned by the list route, which carries run rollups instead of settings. - `id` string, required - `declaredId` string — The suite's declared file identity (`suite.id` in a suite file). Present on file-owned suites; absent on UI-authored suites, which have no declared id and cannot be claimed by `eval run --file`. - `managedBy` 'ci' | 'app' — Where this suite's configuration lives. `ci` means it is owned by a committed suite file or by MCPJam SDK ingest, and the platform REFUSES configuration writes to it — name, settings, environments, schedule, models, skills, execution config and cases — from this API and from the app alike, with `409 CONFLICT` and `details.reason: "CI_OWNED_SUITE_READ_ONLY"`. Running, replaying, comparing and DELETING are unaffected — deleting a suite is not editing one, and it is the only cleanup an SDK-created suite has. To change a CI-owned suite, either edit its file and send that file's `suite.id` as `declaredSuiteId` on the write, or duplicate the suite for an editable copy. `declaredId` alone is not this answer: a suite created by SDK ingest is CI-owned and has no declared id. - `name` string, nullable - `description` string, nullable - `projectId` string, nullable - `environment` object, required — LEGACY server bag, kept as rollback data, plus the suite's sandbox-image pin. Unrelated to `environmentIds`. - `servers` string[] - `computerEnvironment` EvalSuiteComputerEnvironment — The sandbox image a suite's eval runs boot from. - `id` string, required - `name` string, nullable, required — The image's name, resolved for you so a caller can echo back what it set. `null` when the pinned image could not be resolved (deleted, or not visible to this caller). - `executionConfig` object, nullable - `model` string - `systemPrompt` string - `temperature` number - `hosts` object[], required - `id` string, required - `name` string, required - `servers` string[] - `environmentIds` string[] — Attached project environments, in attach order. - `settings` EvalSuiteSettings, required — A suite's grading configuration, as vocabulary 1 spells it. FOUR SEPARATE SETTINGS, and the first two are the ones most often confused: the PASS CRITERION (`minimumAccuracy`, one percentage over the whole run, OR `verdictPolicyDefaults.passThreshold`, a fraction each case must meet over its own iterations — `policy` says which decides this suite), the ITERATION RULE (`minimumIterations`, a floor that RAISES a case's own count, OR `verdictPolicyDefaults.repetitions`, a default that REPLACES it), the evaluators (`matchOptions`, `checks`, `judge`), and the QUALITY GATE, which asks whether a run regressed against a baseline rather than whether it met a bar. The four REQUIRED fields are the ones the DTO always emits: `minimumAccuracy` and `matchOptions` are nullable rather than absent — `null` is the suite's real state (no suite-wide threshold in effect, no match options set), not a stand-in for a default — and `judge` is fully resolved, so it is present even on a suite that configured nothing. - `minimumAccuracy` number, nullable, required — The SUITE-WIDE accuracy threshold, as a PERCENT in [0, 100]: one rate across the whole run must reach it, and individual cases have no threshold of their own. It is NOT `verdictPolicyDefaults.passThreshold` in other units — ten cases, nine always passing and one always failing, passes a 90% suite-wide bar and fails a 0.9 per-case one, so dividing by 100 moves the bar for every suite with more than one case. ALWAYS `null` when `policy` is `v2`, whatever the suite's storage still holds: such a suite is decided per case, and the percent column left behind by a criterion change is read by nothing. Reporting the two side by side would leave a reader to guess which one decides. - `minimumIterations` number, nullable — Suite-level floor on per-case iterations, 1–10: every case runs at least this many times (`max(case.iterations, minimumIterations)`). `null` means no floor — the suite's real state, not a stand-in for 1. PATCH `null` to remove it. - `matchOptions` object, nullable, required - `checks` object[], required - `judge` object, required — Goal-completion judge (LLM as Judge), FULLY RESOLVED: every field is the platform default where the suite set none, so this is what a run on this suite would actually grade with. - `enabled` boolean — Judge is available on the suite. On its own this grades nothing. - `model` string, nullable — Resolved judge model. `null` only from deployments that predate resolution. - `autoRun` boolean — The flag that makes grading HAPPEN: grade every run as it completes. Spends per run. - `threshold` number — Advisory pass threshold, 0–1 (`passed = score >= threshold`). - `severity` 'warn' — Presentation severity on the goal-completion slot. Legal only with an advisory role. - `groundedness` object — Stored groundedness slot, when present. Read-only while execution is unwired; PATCH refuses this key. - `role` 'advisory' - `model` string, nullable - `threshold` number, nullable - `severity` 'warn' - `rubric` object, nullable — The suite's own grading criteria, handed to the judge alongside each case's expected output. `null` when the suite has none. The judge cites `id` in its reasons, so ids are load-bearing; editing them retires the suite's judge calibration. - `criteria` object[], required - `id` string, required - `label` string, required - `description` string - `required` boolean - `verdictPolicyVersion` 2 — The verdict policy this suite's runs are decided under. `2` grades each case against a `passThreshold` FRACTION over its own `repetitions` and decides validity first (an unmeasurable run is `inconclusive`, not failed). ABSENT means legacy: `minimumAccuracy` (a PERCENT) over `max(case.iterations, minimumIterations)`. The two are not convertible, which is why absence is reported rather than defaulted. - `verdictPolicyDefaults` object — Suite defaults a case inherits under PER-CASE GRADING: the pass threshold, the iteration count, and the evidence requirements that let a run be withheld as `inconclusive`. Present only with `verdictPolicyVersion: 2`, and only as a whole — `repetitions` without `passThreshold` cannot answer what a case is graded against. - `repetitions` integer, required — Trials per case unless the case overrides `repetitions`. - `passThreshold` number, required — Fraction of a case's trials that must pass. Never a percent. - `validity` object — When a run counts as measured well enough to decide. DECLARED, not resolved: an omitted field is not "no minimum" but the contract default — `minCompletionRate` 0.8, `maxEvaluatorErrorRate` 0.1, and an omitted `minEligibleTrials` requiring every configured trial attempted plus at least one gradeable trial. The policy a run was actually decided under is on the run's `verdictSummary.validity`. - `minEligibleTrials` integer - `minCompletionRate` number - `maxEvaluatorErrorRate` number - `policy` 'legacy' | 'v2' — Which policy decides this suite's runs, said in one word. `legacy` grades by `minimumAccuracy` (a percent) over `max(case.iterations, minimumIterations)`; `v2` grades each case by `passThreshold` (a fraction) over its own `repetitions` and decides validity first. It is also what says which threshold a PATCH may send — the two are refused together. - `qualityGate` SuiteGatePolicyV1 — Stored quality-gate policy: the conditions a run is failed on for REGRESSING against a baseline run, which is a different question from the pass criteria (`minimumAccuracy` or `verdictPolicyDefaults.passThreshold`). A run can meet its pass criterion and still fail the gate, and the reverse. Every field is optional and an omitted condition is not evaluated at all — it is not a condition set to zero. Numeric fields are FRACTIONS in [0,1] (`maximumPassRateDrop`) or nonnegative milliseconds (`maximumP95LatencyIncreaseMs`); a percent-shaped `3` meaning "3%" is a usage error, not three percentage points. A comparative condition with no resolvable baseline is `non_gateable` rather than passing. - `baseline` union — Where the comparative conditions look for their baseline run. Omitted means there is none, which makes every comparative condition `non_gateable` rather than passing. - object - `kind` 'run', required - `runId` string, required - object - `kind` 'commit_sha', required - `commitSha` string, required - object — RESERVED. The shape exists so a later capability can enable it without forking the schema; every public write refuses it today, and a historical row that already stores it evaluates as `non_gateable` rather than being silently ignored. - `kind` 'previous_completed', required - `maximumPassRateDrop` number — How far the pass rate may fall below the baseline's, as a FRACTION in [0,1]. `0.05` is five percentage points, not five percent of the baseline. - `noDeterministicRegressions` boolean — Fail the run when a scorer that passed deterministically on the baseline does not on this run. - `maximumP95LatencyIncreaseMs` number — How far p95 latency may rise above the baseline's, in milliseconds. - `noGatingScoreErrors` boolean — Fail the run when any required evaluator errored. An evaluator error is not a server defect — it means the run says nothing about the server — so this is a condition rather than a verdict. - `schedule` EvalSuiteSchedule, required — Recurring-run schedule. A schedule fires exactly ONE run, so an environment-based suite pins the environment it runs against. - `enabled` boolean, required - `intervalMinutes` integer, nullable — Minutes between runs. 5 minutes to 7 days. - `environmentId` string, nullable — The single attached environment scheduled runs use. Null on a legacy (non-environment) suite. - `state` 'active' | 'paused_quota' | 'paused_auth' | 'paused_failures' | 'null', nullable — What the schedule is DOING, which `enabled` cannot say. A schedule pauses itself on exhausted quota, on the owner losing access to the suite, and after repeated consecutive failures — all three keep `enabled: true`, so a caller reading only `enabled` reports a healthy automation that has not run in a week. - `createdBy` string, nullable — The user id the schedule runs AS. Scheduled runs use this person's access, and the schedule pauses (`paused_auth`) if they lose it. - `nextDueAt` integer, nullable — Epoch ms of the next due firing, or `null` when nothing is due. - `consecutiveFailures` integer — Consecutive failed firings; resets on the first success. - `revisionNumber` integer, nullable — How many committed edits this suite has had, or `null` on a deployment that does not record revisions. Send it back as `expectedRevisionNumber` on a PATCH to make that edit a compare-and-set. - `createdAt` number, nullable — Epoch milliseconds. - `updatedAt` number, nullable ## Other responses - `400` — Malformed body or parameters. - `401` — Missing, invalid, revoked, or orphaned key (`UNAUTHORIZED`) — or the **target MCP server** needs an OAuth grant (`OAUTH_REQUIRED`), which is a property of the server, not your key. - `403` — Key is valid but not allowed to do this. - `404` — Unknown project, server, or resource. - `409` — The resource is not in a state that accepts this write — a stale `expectedRevision`, a duplicate name, or an environment that cannot currently be launched. The request was well-formed; re-read the resource and retry. - `429` — Per-key rate limit exceeded (60 requests/minute sustained, bursts up to 10). Honor `Retry-After` and back off with jitter. - `500` — Something failed on MCPJam's side. - `502` — Could not connect to the target MCP server. ## Changes - **2026-09-15** `83dc3f970e1a` — 5 info - added the optional property `settings/qualityGate` to the response with the `200` status - the response property `settings/checks` became required for the status `200` - the response property `settings/judge` became required for the status `200` - the response property `settings/matchOptions` became required for the status `200` - …1 more - **2026-09-14** `56b9d1dda7ea` — 1 info - added the new optional `header` request parameter `x-mcpjam-eval-vocabulary` - **2026-09-08** `7c4530b91918` — 3 info - added the new optional `query` request parameter `declaredSuiteId` - added the new optional request property `declaredSuiteId` - added the optional property `managedBy` to the response with the `200` status - **2026-09-07** `ce92b0d7eb75` — 3 info - added the new optional request property `settings/judge/severity` - added the optional property `settings/judge/groundedness` to the response with the `200` status - added the optional property `settings/judge/severity` to the response with the `200` status - **2026-09-04** `e60b7e1653e7` — 12 info - added the new optional request property `expectedRevisionNumber` - added the new optional request property `settings/judge/rubric` - added the new optional request property `settings/passThreshold` - added the new optional request property `settings/repetitions` - …8 more [Full history](https://skmtc.dev/mcpjam/apis/mcpjam-api/changes/projects/:projectId/eval-suites/:suiteId/patch.md) --- [API](https://skmtc.dev/mcpjam/apis/mcpjam-api.md) · [All operations](https://skmtc.dev/mcpjam/apis/mcpjam-api/llms.txt) · [OpenAPI document](https://skmtc.dev/mcpjam/apis/mcpjam-api/revisions/04044ef64a5f?raw)