--- title: "Get one eval suite's configuration" method: GET path: "/projects/{projectId}/eval-suites/{suiteId}" tags: ["Eval runs"] --- # Get one eval suite's configuration `GET /projects/{projectId}/eval-suites/{suiteId}` The suite's full settings — attached environments, hosts, execution config, judge, schedule. The list route returns run rollups instead; this is the shape the edit routes read and write. ## Path parameters - `projectId` string, required - `suiteId` string, required ## Headers - `x-mcpjam-eval-vocabulary` '1' | '2' ## Response `200` The suite. - EvalSuiteDetail — One eval suite's full configuration. Distinct from the `EvalSuite` summary returned by the list route, which carries run rollups instead of settings. - `id` string, required - `declaredId` string — The suite's declared file identity (`suite.id` in a suite file). Present on file-owned suites; absent on UI-authored suites, which have no declared id and cannot be claimed by `eval run --file`. - `managedBy` 'ci' | 'app' — Where this suite's configuration lives. `ci` means it is owned by a committed suite file or by MCPJam SDK ingest, and the platform REFUSES configuration writes to it — name, settings, environments, schedule, models, skills, execution config and cases — from this API and from the app alike, with `409 CONFLICT` and `details.reason: "CI_OWNED_SUITE_READ_ONLY"`. Running, replaying and comparing are unaffected. To change a CI-owned suite, either edit its file and send that file's `suite.id` as `declaredSuiteId` on the write, or duplicate the suite for an editable copy. `declaredId` alone is not this answer: a suite created by SDK ingest is CI-owned and has no declared id. - `name` string, nullable - `description` string, nullable - `projectId` string, nullable - `environment` object, required — LEGACY server bag, kept as rollback data, plus the suite's sandbox-image pin. Unrelated to `environmentIds`. - `servers` string[] - `computerEnvironment` EvalSuiteComputerEnvironment — The sandbox image a suite's eval runs boot from. - `id` string, required - `name` string, nullable, required — The image's name, resolved for you so a caller can echo back what it set. `null` when the pinned image could not be resolved (deleted, or not visible to this caller). - `executionConfig` object, nullable - `model` string - `systemPrompt` string - `temperature` number - `hosts` object[], required - `id` string, required - `name` string, required - `servers` string[] - `environmentIds` string[] — Attached project environments, in attach order. - `settings` EvalSuiteSettings, required — A suite's grading configuration, as vocabulary 1 spells it. FOUR SEPARATE SETTINGS, and the first two are the ones most often confused: the PASS CRITERION (`minimumAccuracy`, one percentage over the whole run, OR `verdictPolicyDefaults.passThreshold`, a fraction each case must meet over its own iterations — `policy` says which decides this suite), the ITERATION RULE (`minimumIterations`, a floor that RAISES a case's own count, OR `verdictPolicyDefaults.repetitions`, a default that REPLACES it), the evaluators (`matchOptions`, `checks`, `judge`), and the QUALITY GATE, which asks whether a run regressed against a baseline rather than whether it met a bar. The four REQUIRED fields are the ones the DTO always emits: `minimumAccuracy` and `matchOptions` are nullable rather than absent — `null` is the suite's real state (no suite-wide threshold in effect, no match options set), not a stand-in for a default — and `judge` is fully resolved, so it is present even on a suite that configured nothing. - `minimumAccuracy` number, nullable, required — The SUITE-WIDE accuracy threshold, as a PERCENT in [0, 100]: one rate across the whole run must reach it, and individual cases have no threshold of their own. It is NOT `verdictPolicyDefaults.passThreshold` in other units — ten cases, nine always passing and one always failing, passes a 90% suite-wide bar and fails a 0.9 per-case one, so dividing by 100 moves the bar for every suite with more than one case. ALWAYS `null` when `policy` is `v2`, whatever the suite's storage still holds: such a suite is decided per case, and the percent column left behind by a criterion change is read by nothing. Reporting the two side by side would leave a reader to guess which one decides. - `minimumIterations` number, nullable — Suite-level floor on per-case iterations, 1–10: every case runs at least this many times (`max(case.iterations, minimumIterations)`). `null` means no floor — the suite's real state, not a stand-in for 1. PATCH `null` to remove it. - `matchOptions` object, nullable, required - `checks` object[], required - `judge` object, required — Goal-completion judge (LLM as Judge), FULLY RESOLVED: every field is the platform default where the suite set none, so this is what a run on this suite would actually grade with. - `enabled` boolean — Judge is available on the suite. On its own this grades nothing. - `model` string, nullable — Resolved judge model. `null` only from deployments that predate resolution. - `autoRun` boolean — The flag that makes grading HAPPEN: grade every run as it completes. Spends per run. - `threshold` number — Advisory pass threshold, 0–1 (`passed = score >= threshold`). - `severity` 'warn' — Presentation severity on the goal-completion slot. Legal only with an advisory role. - `groundedness` object — Stored groundedness slot, when present. Read-only while execution is unwired; PATCH refuses this key. - `role` 'advisory' - `model` string, nullable - `threshold` number, nullable - `severity` 'warn' - `rubric` object, nullable — The suite's own grading criteria, handed to the judge alongside each case's expected output. `null` when the suite has none. The judge cites `id` in its reasons, so ids are load-bearing; editing them retires the suite's judge calibration. - `criteria` object[], required - `id` string, required - `label` string, required - `description` string - `required` boolean - `verdictPolicyVersion` 2 — The verdict policy this suite's runs are decided under. `2` grades each case against a `passThreshold` FRACTION over its own `repetitions` and decides validity first (an unmeasurable run is `inconclusive`, not failed). ABSENT means legacy: `minimumAccuracy` (a PERCENT) over `max(case.iterations, minimumIterations)`. The two are not convertible, which is why absence is reported rather than defaulted. - `verdictPolicyDefaults` object — Suite defaults a case inherits under PER-CASE GRADING: the pass threshold, the iteration count, and the evidence requirements that let a run be withheld as `inconclusive`. Present only with `verdictPolicyVersion: 2`, and only as a whole — `repetitions` without `passThreshold` cannot answer what a case is graded against. - `repetitions` integer, required — Trials per case unless the case overrides `repetitions`. - `passThreshold` number, required — Fraction of a case's trials that must pass. Never a percent. - `validity` object — When a run counts as measured well enough to decide. DECLARED, not resolved: an omitted field is not "no minimum" but the contract default — `minCompletionRate` 0.8, `maxEvaluatorErrorRate` 0.1, and an omitted `minEligibleTrials` requiring every configured trial attempted plus at least one gradeable trial. The policy a run was actually decided under is on the run's `verdictSummary.validity`. - `minEligibleTrials` integer - `minCompletionRate` number - `maxEvaluatorErrorRate` number - `policy` 'legacy' | 'v2' — Which policy decides this suite's runs, said in one word. `legacy` grades by `minimumAccuracy` (a percent) over `max(case.iterations, minimumIterations)`; `v2` grades each case by `passThreshold` (a fraction) over its own `repetitions` and decides validity first. It is also what says which threshold a PATCH may send — the two are refused together. - `qualityGate` SuiteGatePolicyV1 — Stored quality-gate policy: the conditions a run is failed on for REGRESSING against a baseline run, which is a different question from the pass criteria (`minimumAccuracy` or `verdictPolicyDefaults.passThreshold`). A run can meet its pass criterion and still fail the gate, and the reverse. Every field is optional and an omitted condition is not evaluated at all — it is not a condition set to zero. Numeric fields are FRACTIONS in [0,1] (`maximumPassRateDrop`) or nonnegative milliseconds (`maximumP95LatencyIncreaseMs`); a percent-shaped `3` meaning "3%" is a usage error, not three percentage points. A comparative condition with no resolvable baseline is `non_gateable` rather than passing. - `baseline` union — Where the comparative conditions look for their baseline run. Omitted means there is none, which makes every comparative condition `non_gateable` rather than passing. - object - `kind` 'run', required - `runId` string, required - object - `kind` 'commit_sha', required - `commitSha` string, required - object — RESERVED. The shape exists so a later capability can enable it without forking the schema; every public write refuses it today, and a historical row that already stores it evaluates as `non_gateable` rather than being silently ignored. - `kind` 'previous_completed', required - `maximumPassRateDrop` number — How far the pass rate may fall below the baseline's, as a FRACTION in [0,1]. `0.05` is five percentage points, not five percent of the baseline. - `noDeterministicRegressions` boolean — Fail the run when a scorer that passed deterministically on the baseline does not on this run. - `maximumP95LatencyIncreaseMs` number — How far p95 latency may rise above the baseline's, in milliseconds. - `noGatingScoreErrors` boolean — Fail the run when any required evaluator errored. An evaluator error is not a server defect — it means the run says nothing about the server — so this is a condition rather than a verdict. - `schedule` EvalSuiteSchedule, required — Recurring-run schedule. A schedule fires exactly ONE run, so an environment-based suite pins the environment it runs against. - `enabled` boolean, required - `intervalMinutes` integer, nullable — Minutes between runs. 5 minutes to 7 days. - `environmentId` string, nullable — The single attached environment scheduled runs use. Null on a legacy (non-environment) suite. - `state` 'active' | 'paused_quota' | 'paused_auth' | 'paused_failures' | 'null', nullable — What the schedule is DOING, which `enabled` cannot say. A schedule pauses itself on exhausted quota, on the owner losing access to the suite, and after repeated consecutive failures — all three keep `enabled: true`, so a caller reading only `enabled` reports a healthy automation that has not run in a week. - `createdBy` string, nullable — The user id the schedule runs AS. Scheduled runs use this person's access, and the schedule pauses (`paused_auth`) if they lose it. - `nextDueAt` integer, nullable — Epoch ms of the next due firing, or `null` when nothing is due. - `consecutiveFailures` integer — Consecutive failed firings; resets on the first success. - `revisionNumber` integer, nullable — How many committed edits this suite has had, or `null` on a deployment that does not record revisions. Send it back as `expectedRevisionNumber` on a PATCH to make that edit a compare-and-set. - `createdAt` number, nullable — Epoch milliseconds. - `updatedAt` number, nullable ## Other responses - `401` — Missing, invalid, revoked, or orphaned key (`UNAUTHORIZED`) — or the **target MCP server** needs an OAuth grant (`OAUTH_REQUIRED`), which is a property of the server, not your key. - `403` — Key is valid but not allowed to do this. - `404` — Unknown project, server, or resource. - `429` — Per-key rate limit exceeded (60 requests/minute sustained, bursts up to 10). Honor `Retry-After` and back off with jitter. - `500` — Something failed on MCPJam's side. - `502` — Could not connect to the target MCP server. ## Changes - **2026-09-15** `83dc3f970e1a` — 5 info - added the optional property `settings/qualityGate` to the response with the `200` status - the response property `settings/checks` became required for the status `200` - the response property `settings/judge` became required for the status `200` - the response property `settings/matchOptions` became required for the status `200` - …1 more - **2026-09-14** `56b9d1dda7ea` — 1 info - added the new optional `header` request parameter `x-mcpjam-eval-vocabulary` - **2026-09-08** `7c4530b91918` — 1 info - added the optional property `managedBy` to the response with the `200` status - **2026-09-07** `ce92b0d7eb75` — 2 info - added the optional property `settings/judge/groundedness` to the response with the `200` status - added the optional property `settings/judge/severity` to the response with the `200` status - **2026-09-04** `e60b7e1653e7` — 7 info - added the optional property `revisionNumber` to the response with the `200` status - added the optional property `schedule/consecutiveFailures` to the response with the `200` status - added the optional property `schedule/createdBy` to the response with the `200` status - added the optional property `schedule/nextDueAt` to the response with the `200` status - …3 more [Full history](https://skmtc.dev/mcpjam/apis/mcpjam-api/changes/projects/:projectId/eval-suites/:suiteId/get.md) --- [API](https://skmtc.dev/mcpjam/apis/mcpjam-api.md) · [All operations](https://skmtc.dev/mcpjam/apis/mcpjam-api/llms.txt) · [OpenAPI document](https://skmtc.dev/mcpjam/apis/mcpjam-api/revisions/ae334d76415e?raw)