--- title: "List run iterations" method: GET path: "/projects/{projectId}/eval-runs/{runId}/iterations" tags: ["Eval runs"] --- # List run iterations `GET /projects/{projectId}/eval-runs/{runId}/iterations` Per-iteration results: actual tool calls, structured token usage, and latency. Cursor-paginated. ## Path parameters - `projectId` string, required - `runId` string, required ## Query parameters - `limit` integer - `cursor` string ## Headers - `x-mcpjam-eval-vocabulary` '1' | '2' ## Response `200` One page of iterations. - EvalIterationPage - `items` EvalIteration[], required - `id` string, required - `testCaseId` string, nullable — The STORED case row's database id. Changes if the case is recreated and means nothing outside this deployment; see `caseId` for the declared one. - `caseId` string — The case's SDK-DECLARED id, read from the iteration's frozen `testCaseSnapshot` — the durable, author-chosen identity that survives the case row being recreated. ABSENT on a UI-authored case (which never declared one) and on runs predating declared ids; absence is not an error. Distinct from `testCaseId`, which is this deployment's row id, and NOT a join key into `verdictSummary.cases[].caseId`, which is a separately encoded identity the platform mints from whichever spelling the run knew. - `title` string, nullable - `iterationNumber` integer, required - `status` 'pending' | 'running' | 'completed' | 'failed' | 'cancelled' | 'timed_out' | 'setup_failed' | 'skipped', required — LIFECYCLE, not verdict: how far the trial got, never how it graded. A trial that ran normally and graded badly is `completed` with `result: "failed"`, so reading `status === "failed"` as "the case failed" counts harness noise as server defects. `failed` is an execution failure, `setup_failed` means the environment never came up, `timed_out` means the budget expired, `skipped` means it was deliberately not run, `cancelled` means it was withdrawn mid-flight (and is excluded from validity denominators). Treat an unrecognised value as non-terminal-unknown rather than as a failure. - `result` 'passed' | 'failed' | 'null', nullable — Task verdict for the trial, independent of `status`. `null` until the trial is graded — including on a terminal trial that never produced a gradeable outcome (`setup_failed`, `skipped`). - `model` string, nullable - `provider` string, nullable - `startedAt` number, nullable — Epoch milliseconds. - `durationMs` number, nullable — Wall-clock duration; `null` until terminal. - `tokensUsed` number, nullable - `usage` object, nullable — Structured token usage (input/output/cached/reasoning) when available, plus `estimatedCostUsd` and the `costBasis` it was priced from. `estimatedCostUsd` ABSENT means no cost was observed — never that the trial was free. `costBasis.reason` says which: `no_pricing` (not an MCPJam-billed model, e.g. your own API key), `harness_mixed_models` (a harness turn mixes models and carries no per-model split, so its cost arrives with billed attribution instead), or `no_tokens`. `costBasis.source` is `gateway_pricing` when MCPJam priced its own token counts, and `sdk_runner` when the figure came from your runner — real, but neither computed nor verified by MCPJam. - `actualToolCalls` object[], required - `expectedToolCalls` object[], required - `error` string, nullable - `scores` ScoreResult[], nullable — Per-scorer verdicts for this iteration. `null` when the run predates scoring, or when the stored payload failed validation at the boundary — partially-trusted score data is never projected. - `scorerId` string, required - `scorerVersion` string, required - `definitionHash` string, required — Joins this result to its definition in `evaluationConfig.definitions`. - `status` 'scored' | 'error' | 'skipped' | 'not_applicable', required — `error` is never a low score — a crashed judge is not a disagreeing judge. `not_applicable` never gates and is excluded from aggregation denominators, which is what distinguishes it from `skipped`. - `value` number — Present only when `status` is `scored`. - `passThreshold` number, required - `passed` boolean — Derived as `value >= passThreshold`; present only when `status` is `scored`. - `rationale` string - `evidence` string[] - `deterministic` boolean, required - `model` string - `promptHash` string - `error` string — Present only when `status` is `error`. - `scope` object — Absent means case-level; `{kind:"turn",promptIndex}` means the check was authored on a single turn. - `evaluationConfig` EvaluationConfigSnapshot — The scorer definitions a run graded with, plus their hash. Changing scorer configuration changes this hash but NOT the case identity — a threshold edit must not fork a scenario's history. - `hash` string, required — `evaluationConfigHash` over the resolved definitions. Order-independent — the hash sorts internally. - `definitions` ResolvedScoreDefinition[], required - `scorerId` string, required - `idSource` 'explicit' | 'generated' | 'platform', required — `generated` ids are positional and UNSTABLE across config edits; only `explicit` ids may be referenced by a gate policy or tracked across runs. `platform` marks a scorer the platform owns rather than the suite author — the hosted judge — whose id is stable but not author-editable. - `scorerVersion` string, required - `implementationHash` string, required — Digest of what the scorer actually does — the canonicalized predicate, or the judge prompt plus template version. Required so two judges with different prompts cannot hash identically. - `label` string — Presentation only; deliberately excluded from the hash. - `deterministic` boolean, required - `passThreshold` number, required - `role` 'gating' | 'advisory' | 'required', required — Only required scorers decide the iteration's verdict. `required` is the canonical spelling and `gating` its legacy one; a response uses `gating` unless the request sent `x-mcpjam-eval-vocabulary: 2`. - `onError` 'fail' | 'ignore', required — What an `error` status does to a gating iteration. Defaults to `fail`. - `onSkipped` 'fail' | 'ignore', required — What a `skipped` status does to a gating iteration. Separate from `onError`: a crashed judge and a judge that never ran are different failures. - `model` string - `scope` object - `scoreIntegrity` 'score_integrity_invalid' | 'null', nullable — Set when the backend downgraded this iteration's verdict because its gating score evidence was malformed. - `stageResults` StageResultRow[] — The user-value chain rows for this trial, in chain order, from a derivation that validated. - `stage` 'connection' | 'discovery' | 'selection' | 'call' | 'response' | 'userValue', required — Which link of the user-value chain this row is a verdict on. The order is NORMATIVE: a run walks the six in sequence, and `notReached` is derived from position — every stage after the first failed one never ran. `connection` — the server was reachable and the session initialized. `discovery` — its tools and resources were listed and readable. `selection` — the model chose the right tool for the request. `call` — the call was made with usable arguments. `response` — the server returned data the model could use. `userValue` — the user's actual request was satisfied. - `state` 'passed' | 'failed' | 'notReached' | 'notMeasured' | 'notApplicable', required — What this stage did. The three non-verdicts stay three different facts on purpose: collapsing them is how "we never checked" gets read as "it passed". `passed` — measured, and it passed. `failed` — measured, and it failed. `notReached` — it never ran (an earlier stage failed), so there was nothing to decide. `notMeasured` — this run captured nothing that could decide it, so the stage is not measured. `notApplicable` — the stage is not applicable to this case at all. - `reason` 'noSpanChannel' | 'noEvidenceCaptured' | 'matchVerdictUnavailable' | 'traceAbsent' | 'executorEmitsNoSpans' | 'blockedByPolicy' | 'evaluatorError' | 'providerError' | 'setupAborted' | 'connectFailed' | 'toolsListFailed' | 'egressUnverified' | 'lifecycleStopped' | 'notAuthored' | 'earlierStageFailed' | 'missingToolCall' | 'unexpectedToolCall' | 'argumentMismatch' | 'toolError' | 'protocolError' | 'renderFailed' | 'predicateFailed' | 'observed' | 'impliedByLaterEvidence' | 'judgeObserved' | 'judgePartial' | 'judgeFailed' | 'judgePending' | 'judgeNotRequested' — Why this stage landed where it did. A CLOSED vocabulary — render what arrives, never widen it. Each entry completes "…because ". `noSpanChannel` — this run captures no evidence channel for that stage. `noEvidenceCaptured` — nothing eligible for that stage was captured. `matchVerdictUnavailable` — extra tool calls were captured but the run did not report whether its match options tolerate them. `traceAbsent` — the iteration recorded no trace. `executorEmitsNoSpans` — the executor emitted no spans. `blockedByPolicy` — a policy blocked the run before it could be measured. `evaluatorError` — the evaluator itself failed, so the run says nothing about the server. `providerError` — the model provider failed the call, so this stage was never measured. `setupAborted` — the environment was never prepared, so the test never began. `connectFailed` — the configured server was reached and initialize failed there. `toolsListFailed` — initialize succeeded and listing tools failed. `egressUnverified` — the connection failed with no evidence that our own network egress works. `lifecycleStopped` — the run was stopped mid-flight. `notAuthored` — the case asserts nothing this stage could decide. `earlierStageFailed` — an earlier stage failed. `missingToolCall` — an expected tool call was never made. `unexpectedToolCall` — a tool call was made that the case did not expect. `argumentMismatch` — the call arguments did not match what the case expects. `toolError` — the server reported a tool error. `protocolError` — the call never produced a result. `renderFailed` — the widget did not render. `predicateFailed` — an assertion on the result did not hold. `observed` — the evidence was inspected and the stage held. `impliedByLaterEvidence` — a later stage's success implies it. `judgeObserved` — the LLM judge scored at or above the threshold. `judgePartial` — the LLM judge scored inside the partial band — at or above the floor, below the threshold. `judgeFailed` — the LLM judge scored below the partial floor. `judgePending` — an LLM judge verdict is owed and has not arrived. `judgeNotRequested` — no LLM judge verdict was ever owed. - `evidence` object - `spanIds` string[] - `promptIndexes` integer[] - `predicateReasons` string[] - `firstFailedStage` 'connection' | 'discovery' | 'selection' | 'call' | 'response' | 'userValue' — Where the chain stopped — A LOCATION, NOT A CAUSE. It names the first link that failed, never why it failed: the why is `reason` on that stage's row (see `StageResultRow.reason`), and neither this nor the `failureCategory` bucket on its own authorizes proposing a change to the server. Absent when nothing failed — and absent on a trial that never reached a stage at all, such as a setup abort or an evaluator error, which still carries a `failureCategory`. `connection` — the server was reachable and the session initialized. `discovery` — its tools and resources were listed and readable. `selection` — the model chose the right tool for the request. `call` — the call was made with usable arguments. `response` — the server returned data the model could use. `userValue` — the user's actual request was satisfied. - `failureCategory` 'setup' | 'metadata' | 'selection' | 'arguments' | 'serverData' | 'userValue' | 'evaluator' — The bucket this trial is grouped under — A BUCKET, NOT A DIAGNOSIS. Present without `firstFailedStage` for a trial that never reached a stage. `setup` — setup: the harness or environment never got to the test. `metadata` — tool metadata: tool names, descriptions or schemas misled the model. `selection` — tool selection: the model picked the wrong tool, or none. `arguments` — call arguments: the right tool, called wrongly. `serverData` — server data: the server answered, with data the model could not use. `userValue` — user value: everything mechanical worked and the user still was not served. `evaluator` — evaluator: the grader itself failed, so the run says nothing about the server — never folded into the others, because counting a broken judge as a server defect poisons every rate derived from it. - `stageAnalyzerVersion` integer - `stageResultsUnverified` true - `frictionSignals` EvalTrialFrictionSignals — Observable patterns in one trial's tool calls. REPORT-ONLY: nothing here decided the trial's `result`, and every pattern has a benign reading. `state` is `notMeasured` when the trial could not be looked at at all; `identifierSignals.state` is separate because a trial whose tool results were not retained still has measurable retries but cannot be asked whether an identifier went unused. An ABSENT document means the trial predates the measurement — never read it as zero. - `version` 1, required — Derivation semantics version. Bumped when a counting RULE changes, even if the shape does not. - `state` 'measured' | 'notMeasured', required - `notMeasuredReason` 'noToolCalls' | 'resultsUnavailable' | 'orderingUnknown' | 'evidenceIncomplete' | 'truncated' — Present only when `state` is `notMeasured`. `noToolCalls` — no tool calls to look at. `resultsUnavailable` — tool results were not retained. `orderingUnknown` — the calls cannot be placed in a causal order. `evidenceIncomplete` — the evidence for this iteration has a known hole. `truncated` — too many tool calls to measure. - `callCount` integer, required - `resultAvailableCount` integer, required - `timedCallCount` integer, required - `identifierSignals` object, required — Whether the two identifier-dependent kinds were measured at all. `notMeasured` here beside `state: measured` above is the normal shape for a run whose tool results were not retained: the retries were measured, the identifiers were never looked for. - `state` 'measured' | 'notMeasured', required - `reason` 'noToolCalls' | 'resultsUnavailable' | 'orderingUnknown' | 'evidenceIncomplete' | 'truncated' — Why the identifier kinds were not measured. `noToolCalls` — no tool calls to look at. `resultsUnavailable` — tool results were not retained. `orderingUnknown` — the calls cannot be placed in a causal order. `evidenceIncomplete` — the evidence for this iteration has a known hole. `truncated` — too many tool calls to measure. - `signals` EvalFrictionSignal[], required — Ordered by the call that made each pattern observable. Empty on a `measured` document means nothing fired; empty on a `notMeasured` one means nothing was looked for. - `kind` 'identifierSurfacedUnused' | 'searchRepeatedAfterIdentifier' | 'identicalRetry' | 'changedRetry' | 'paginationContinuation', required — Which observable pattern this is. A CLOSED vocabulary; a reader that does not know a member must not guess at it. Every member has a benign reading, so none of them is a verdict about the trial. `identifierSurfacedUnused` — Identifiers surfaced, none used later: a result carried identifiers and no later call passed one, which is also what a search that already answered the question looks like. `searchRepeatedAfterIdentifier` — Same tool searched again after identifiers: the tool that surfaced them was called again with a different question, which is also what a sensible refinement looks like. `identicalRetry` — Repeated with identical arguments: the same call ran twice byte for byte, which is what recovery after a transient error looks like. `changedRetry` — Same tool called again with changed arguments: ordinary exploration, and most useful work looks like this. `paginationContinuation` — Pagination continued: only the pagination keys changed, which is a cursor doing its job and is never counted as a repeat. - `toolName` string, required - `toolCallId` string - `informationCallIndex` integer - `observedAtCallIndex` integer - `callIndex` integer - `priorCallIndex` integer - `afterError` boolean — Whether the call this one repeats returned an error. - `identifierKeyPaths` string[] — Where in the result the identifiers were found (`results[].id`). KEY PATHS ONLY — an identifier VALUE never leaves the deployment. - `identifierCount` integer - `laterCallCount` integer - `repeatCallIndexes` integer[] - `paginationKeys` string[] - `frictionSignalsUnverified` true — The server stored a friction document that failed validation. The signals themselves are withheld rather than partially trusted. - `suspectedConditionVerdict` EvalSuspectedConditionVerdict — Which server-controlled condition is SUSPECTED of contributing to one observed friction pattern, from an advisory per-trial judge. REPORT-ONLY and NEVER a claim about cause: nothing here entered the trial’s `result`, its stage chain or its `failureCategory`, and only a controlled rewrite that changes the suspected response and holds the rest comparable could establish causation. ABSENT means no judge ran — the trial predates it, nothing was flagged, or the deployment has it off. - `status` 'scored' | 'skipped' | 'error', required — `scored` — a verdict was produced. `skipped` — the judge did not reach this trial; `reason` says why. `error` — the pass threw for this trial. A skipped or errored trial carries NO condition, so “we never looked” is never rendered as “we looked and could not say”. - `reason` 'spendBlocked' | 'traceIncomplete' | 'noEvidence' | 'cap' — Present only when `status` is `skipped`. `spendBlocked` — the internal-LLM spend gate refused the call. `traceIncomplete` — a referenced trace blob could not be read. `noEvidence` — the judge’s window could not be built, most often because the information call’s result never reached the trace. `cap` — the run had more flagged trials than the per-run cap. - `condition` 'unclear' | 'idBuriedInPayload' | 'idNameCollision' | 'missingQueryEcho' | 'silentTruncation' | 'ambiguousErrorSemantics' | 'missingUnits' | 'responseWasClear' | 'descriptionMisleading' — Which server-controlled condition is SUSPECTED of contributing. A CLOSED vocabulary. SUSPECTED, not established: the judge saw one window of one trial and named a plausible contributor, and only a controlled rewrite could turn that into a claim about cause. `unclear` — could not attribute: the judge could not identify a condition from what it was shown, which is also where a low-confidence or unsupported verdict is demoted to. `idBuriedInPayload` — identifier buried in payload: the identifier a later call needed was not where a caller would look for it. `idNameCollision` — identifier name collision: two different identifiers share a name, or one name means different things across tools. `missingQueryEcho` — response does not echo the query: a caller cannot tell a narrowed result from a fresh one. `silentTruncation` — response truncated without saying so. `ambiguousErrorSemantics` — error text does not say whether to retry. `missingUnits` — value has no stated unit. `responseWasClear` — the response was clear: the server's output does not explain the pattern. This is the honest negative and a real answer. `descriptionMisleading` — tool description points the wrong way. - `confidence` 'low' | 'medium' | 'high' — How sure the judge was. `low` — low confidence: never reaches a reader as a named condition, because a low-confidence verdict is demoted to `unclear` before it is persisted. `medium` — medium confidence. `high` — high confidence. - `remediation` string — ONE sentence naming a lever the SERVER OWNER controls. Absent for `unclear` and `responseWasClear`, which name no server problem to remediate, and absent from any verdict whose remediation named no server lever — that verdict is demoted to `unclear`. - `evidence` object — Where in the window the judge was pointing. Dropped rather than guessed when it fell outside the window or named the wrong tool; the verdict survives, the pointer does not. - `callIndex` integer, required - `toolName` string, required - `fieldPath` string - `gradingKey` string, required - `signalKind` string, required — The friction signal that triggered the judge. - `informationCallIndex` integer, required — Where the information was surfaced. The judge saw calls 0 through `observedAtCallIndex` and nothing after. - `observedAtCallIndex` integer, required — The call that made the pattern observable, and the judge’s evidence bound. - `judgeTemplateVersion` integer, required - `judgeTemplateHash` string, required - `model` string, required - `generatedAt` integer, required - `suspectedConditionUnverified` true — The server stored a suspected-condition verdict that failed validation. The verdict is withheld rather than partially trusted. - `execution` ExecutionRecord — What a run or turn actually ran on: requested vs resolved selection, harness runtime, the settings the request was sent with, every routing attempt, the provider-reported model, and any deviation from what was requested. Names a connection, never a key. ABSENT on rows recorded before the record existed; render that as not recorded. - `requested` object, required — The saved model selection the run asked for. `source: "legacy"` means the stored choice named only a model id, so its source was inferred at run time. - `source` 'hosted' | 'org' | 'local' | 'legacy', required - `modelId` string, required - `resolved` object, required - `rail` string, required — `gateway` (Vercel AI Gateway, MCPJam key), `openrouter` (OpenRouter, MCPJam key), `orgCloud` (an organization provider connection), `local` (a key on the user's own machine). Read leniently: a newer deployment may add a rail. - `wireModelId` string, required - `connectionRef` object — Which connection served a non-hosted selection (`orgProvider` by id, or `localProvider` by provider key). - `nativeModelId` string - `offering` object, required — The provider offering, snapshotted at run start so a connection deleted or re-keyed since cannot change the answer. - `rail` string, required — `gateway` (Vercel AI Gateway, MCPJam key), `openrouter` (OpenRouter, MCPJam key), `orgCloud` (an organization provider connection), `local` (a key on the user's own machine). Read leniently: a newer deployment may add a rail. - `providerKey` string, required - `connectionLabel` string - `credentialVersion` number — Non-secret version of the connection's credential (its last update time). - `nativeModelId` string - `harness` object - `id` string, required - `runtimeVersion` string, required - `effectiveSettings` object, required - `reasoningEffort` string - `temperature` number - `maxOutputTokens` integer, required — `0` means no ceiling was sent and the provider's default applied. - `attempts` object[], required - `rail` string, required — `gateway` (Vercel AI Gateway, MCPJam key), `openrouter` (OpenRouter, MCPJam key), `orgCloud` (an organization provider connection), `local` (a key on the user's own machine). Read leniently: a newer deployment may add a rail. - `wireModelId` string, required - `outcome` 'ok' | 'error', required - `code` string - `at` number, required - `upstreamModel` string — The model id the provider reported serving, when it reported one. - `deviation` object - `kind` 'provider_fallback' | 'model_substitution' | 'harness_substitution', required - `reason` string, required - `nextCursor` string — Opaque cursor for the next page. Omitted on the last page. ## Other responses - `401` — Missing, invalid, revoked, or orphaned key (`UNAUTHORIZED`) — or the **target MCP server** needs an OAuth grant (`OAUTH_REQUIRED`), which is a property of the server, not your key. - `403` — Key is valid but not allowed to do this. - `404` — Unknown project, server, or resource. - `429` — Per-key rate limit exceeded (60 requests/minute sustained, bursts up to 10). Honor `Retry-After` and back off with jitter. - `500` — Something failed on MCPJam's side. - `502` — Could not connect to the target MCP server. ## Changes - **2026-09-25** `a15a9864fb85` — 1 info - added the optional property `items/items/execution` to the response with the `200` status - **2026-09-14** `56b9d1dda7ea` — 2 warning, 1 info - added the new `platform` enum value to the `items/items/evaluationConfig/oneOf[#/components/schemas/EvaluationConfigSnapshot]/definitions/items/idSource` response property for the response status `200` - added the new `required` enum value to the `items/items/evaluationConfig/oneOf[#/components/schemas/EvaluationConfigSnapshot]/definitions/items/role` response property for the response status `200` - added the new optional `header` request parameter `x-mcpjam-eval-vocabulary` - **2026-09-12** `de57cfcdd824` — 1 info - added the non-success response with the status `502` - **2026-09-08** `66613c6fd3a9` — 4 info - added the optional property `items/items/frictionSignals` to the response with the `200` status - added the optional property `items/items/frictionSignalsUnverified` to the response with the `200` status - added the optional property `items/items/suspectedConditionUnverified` to the response with the `200` status - added the optional property `items/items/suspectedConditionVerdict` to the response with the `200` status - …earlier changes not shown [Full history](https://skmtc.dev/mcpjam/apis/mcpjam-api/changes/projects/:projectId/eval-runs/:runId/iterations/get.md) --- [API](https://skmtc.dev/mcpjam/apis/mcpjam-api.md) · [All operations](https://skmtc.dev/mcpjam/apis/mcpjam-api/llms.txt) · [OpenAPI document](https://skmtc.dev/mcpjam/apis/mcpjam-api/revisions/a15a9864fb85?raw)