---
title: "Compare a run against a baseline"
method: GET
path: "/projects/{projectId}/eval-runs/{runId}/compare"
tags: ["Eval runs"]
---

# Compare a run against a baseline

`GET /projects/{projectId}/eval-runs/{runId}/compare`

Compare this run against a baseline run: per-case status (`regressed`, `fixed`, `new_case`, `removed_case`, `changed`), per-scorer pass-rate and mean deltas from the evaluation contract, and whether the evaluation config changed. Omit `baseRunId` to compare against the nearest earlier **completed** run in the same suite. Returns `404` with `details.reason` = `BASELINE_NOT_FOUND` when there is no comparable predecessor — that is an incomplete comparison, not a failing one. A scorer whose `definitionChanged` is `true` was graded by a different definition on each side, so its delta is not a regression.

## Path parameters

- `projectId` string, required
- `runId` string, required

## Query parameters

- `baseRunId` string
- `baseCommitSha` string

## Headers

- `x-mcpjam-eval-vocabulary` '1' | '2'

## Response `200`

The comparison.

- EvalRunCompare
  - `suite` object, required
    - `id` string, required
    - `name` string, required
  - `baseline` object, required
    - `policy` 'previous_completed' | 'previous_completed_same_environment' | 'run' | 'commit_sha', required — How the baseline was chosen. `commit_sha` means it was resolved from the requested source SHA.
    - `baseRunId` string, required
    - `baseCommitSha` string — The pinned source SHA, echoed back for the `commit_sha` policy only.
    - `matchCount` integer — Present ONLY when uniqueness could not be established - the SHA matched several eligible runs, or the bounded lookup saturated. Absent means the match was unambiguous; do not default it to 1.
    - `matchCountTruncated` boolean — `matchCount` is a FLOOR, not a total - including when it reads 1. Read it together with `matchCount`; a count without this flag would assert a uniqueness nobody checked.
  - `baseRun` EvalRunCompareSide, required
    - `id` string, required
    - `runNumber` integer, required
    - `result` string, required
    - `createdAt` integer, required
    - `completedAt` integer, nullable, required
    - `summary` object, nullable, required
      - `total` integer, required
      - `passed` integer, required
      - `failed` integer, required
      - `passRate` number, required
    - `environment` object
      - `id` string, required
      - `name` string, nullable, required
    - `effectiveModelId` string — Model the compared run actually executed with.
    - `client` object
      - `name` string, required
    - `modelSource` 'client_default' | 'override' | 'case'
  - `compareRun` EvalRunCompareSide, required
    - `id` string, required
    - `runNumber` integer, required
    - `result` string, required
    - `createdAt` integer, required
    - `completedAt` integer, nullable, required
    - `summary` object, nullable, required
      - `total` integer, required
      - `passed` integer, required
      - `failed` integer, required
      - `passRate` number, required
    - `environment` object
      - `id` string, required
      - `name` string, nullable, required
    - `effectiveModelId` string — Model the compared run actually executed with.
    - `client` object
      - `name` string, required
    - `modelSource` 'client_default' | 'override' | 'case'
  - `passSummary` object, required — Run-summary counters. Named `passSummary`, not `scores`, so it cannot be confused with `scoreContract` — the two answer different questions.
    - `passRatePercent` NumericDiff, required — A base/compare pair with its delta. Rate-valued instances are FRACTIONS unless the field name ends in `Percent`.
      - `base` number, nullable, required
      - `compare` number, nullable, required
      - `delta` number, nullable, required
      - `percentDelta` number, nullable, required
    - `total` NumericDiff, required — A base/compare pair with its delta. Rate-valued instances are FRACTIONS unless the field name ends in `Percent`.
      - `base` number, nullable, required
      - `compare` number, nullable, required
      - `delta` number, nullable, required
      - `percentDelta` number, nullable, required
    - `passed` NumericDiff, required — A base/compare pair with its delta. Rate-valued instances are FRACTIONS unless the field name ends in `Percent`.
      - `base` number, nullable, required
      - `compare` number, nullable, required
      - `delta` number, nullable, required
      - `percentDelta` number, nullable, required
    - `failed` NumericDiff, required — A base/compare pair with its delta. Rate-valued instances are FRACTIONS unless the field name ends in `Percent`.
      - `base` number, nullable, required
      - `compare` number, nullable, required
      - `delta` number, nullable, required
      - `percentDelta` number, nullable, required
  - `metrics` object, required
    - `wallDurationMs` NumericDiff, required — A base/compare pair with its delta. Rate-valued instances are FRACTIONS unless the field name ends in `Percent`.
      - `base` number, nullable, required
      - `compare` number, nullable, required
      - `delta` number, nullable, required
      - `percentDelta` number, nullable, required
    - `totalTokens` NumericDiff, required — A base/compare pair with its delta. Rate-valued instances are FRACTIONS unless the field name ends in `Percent`.
      - `base` number, nullable, required
      - `compare` number, nullable, required
      - `delta` number, nullable, required
      - `percentDelta` number, nullable, required
    - `estimatedCostUsd` NumericDiff, required — A base/compare pair with its delta. Rate-valued instances are FRACTIONS unless the field name ends in `Percent`.
      - `base` number, nullable, required
      - `compare` number, nullable, required
      - `delta` number, nullable, required
      - `percentDelta` number, nullable, required
    - `costCoverage` CostCoverage — How many iterations on each side of a comparison actually contributed a cost. Cost is stamped per iteration and is absent whenever the model was not MCPJam-billed (your own API key), the run used a harness (which mixes models within a turn), or the trial reported no tokens. Summing the priced iterations and skipping the rest yields a figure indistinguishable from a complete one — so a drop in `estimatedCostUsd` is only a cost signal when coverage is full on both sides. A CI gate should treat partial coverage as non-gateable rather than as a pass or a fail.
      - `base` CostCoverageSide, required
        - `costed` integer, required — Iterations that contributed a cost.
        - `total` integer, required — Iterations on this side.
      - `compare` CostCoverageSide, required
        - `costed` integer, required — Iterations that contributed a cost.
        - `total` integer, required — Iterations on this side.
  - `skills` object, nullable — Which skills changed between the two runs — the configuration attribution that usually explains the case-level differences. `null` when NEITHER run recorded pinned skills (an empty section would instead claim no skills were involved). Absent when the deployment predates skill attribution, so clients must tolerate all three states.
    - `base` EvalRunCompareSkillsSide, required
      - `excluded` boolean, required — This run deliberately ran with skills disabled — distinct from a run that simply pinned none.
      - `count` integer, required — How many skills this run pinned.
    - `compare` EvalRunCompareSkillsSide, required
      - `excluded` boolean, required — This run deliberately ran with skills disabled — distinct from a run that simply pinned none.
      - `count` integer, required — How many skills this run pinned.
    - `changes` EvalRunCompareSkillChange[], required — Added, removed and changed skills only, changed first. Unchanged skills are counted, not listed.
      - `key` string, required — Stable match key; opaque, safe for list keys and dedupe.
      - `name` string, required
      - `modelRef` string — Namespaced runtime address for a plugin-channel skill.
      - `channels` string[], required
      - `kind` 'added' | 'removed' | 'changed', required
      - `renamedFrom` string — Present when the skill was renamed between the runs; it is still matched as ONE skill by its logical id.
      - `base` EvalRunCompareSkillSide — One skill's identity on one side of the comparison.
        - `contentHash` string, required
        - `aggregateHash` string — Complete-artifact hash; present only when supporting files diverge it from `contentHash`.
        - `versionNumber` integer — Authored-skill revision, when the run recorded one.
        - `serverSkillVersionNumber` integer — MCP-captured revision, when the run recorded one.
      - `compare` EvalRunCompareSkillSide — One skill's identity on one side of the comparison.
        - `contentHash` string, required
        - `aggregateHash` string — Complete-artifact hash; present only when supporting files diverge it from `contentHash`.
        - `versionNumber` integer — Authored-skill revision, when the run recorded one.
        - `serverSkillVersionNumber` integer — MCP-captured revision, when the run recorded one.
      - `versionDelta` string — Human-readable revision move (`v3 → v4`), present only when BOTH sides recorded a number. A change with no delta is a real content change whose revisions are unknown, not a smaller change.
    - `unchangedCount` integer, required
  - `scoreContract` object, required
    - `base` ScoreContractSide, required
      - `evaluationConfigHash` string, nullable, required
      - `scoreIntegrity` 'valid' | 'invalid' | 'null', nullable, required — `null` means no verdict was produced. A gate must treat it exactly like `invalid` — absent evidence is not valid evidence.
      - `scoredIterations` integer, required
      - `quarantinedIterations` integer, required — Iterations with at least one row that failed to verify at ingest. Counted here, and excluded from every rate.
    - `compare` ScoreContractSide, required
      - `evaluationConfigHash` string, nullable, required
      - `scoreIntegrity` 'valid' | 'invalid' | 'null', nullable, required — `null` means no verdict was produced. A gate must treat it exactly like `invalid` — absent evidence is not valid evidence.
      - `scoredIterations` integer, required
      - `quarantinedIterations` integer, required — Iterations with at least one row that failed to verify at ingest. Counted here, and excluded from every rate.
    - `evaluationConfigChanged` boolean, required
    - `scorers` ScoreContractScorer[], required
      - `scorerId` string, required
      - `gating` boolean, required
      - `deterministic` boolean, required
      - `definitionChanged` boolean, required — The same scorer id was graded under a different definition hash on each side. Its delta is NOT a regression — the two runs did not measure the same thing.
      - `passRate` NumericDiff, required — A base/compare pair with its delta. Rate-valued instances are FRACTIONS unless the field name ends in `Percent`.
        - `base` number, nullable, required
        - `compare` number, nullable, required
        - `delta` number, nullable, required
        - `percentDelta` number, nullable, required
      - `meanValue` NumericDiff, required — A base/compare pair with its delta. Rate-valued instances are FRACTIONS unless the field name ends in `Percent`.
        - `base` number, nullable, required
        - `compare` number, nullable, required
        - `delta` number, nullable, required
        - `percentDelta` number, nullable, required
      - `errorCount` object, required
        - `base` integer, required
        - `compare` integer, required
  - `cases` EvalRunCompareCase[], required
    - `caseKey` string, required
    - `title` string, required
    - `status` 'unchanged_passed' | 'unchanged_failed' | 'regressed' | 'fixed' | 'new_case' | 'removed_case' | 'changed', required
    - `configChanged` boolean, required — The scenario's own config (prompt, steps, expectations) changed.
    - `evaluationConfigChanged` boolean, required — This case's evaluation config changed.
    - `scoreDeltas` CaseScoreDelta[], required
      - `scorerId` string, required
      - `gating` boolean, required
      - `deterministic` boolean, required
      - `definitionChanged` boolean, required
      - `base` CaseScoreSide, required
        - `status` 'scored' | 'error' | 'skipped' | 'not_applicable', required
        - `value` number, nullable, required
        - `passed` boolean, nullable, required
      - `compare` CaseScoreSide, required
        - `status` 'scored' | 'error' | 'skipped' | 'not_applicable', required
        - `value` number, nullable, required
        - `passed` boolean, nullable, required
      - `value` NumericDiff, required — A base/compare pair with its delta. Rate-valued instances are FRACTIONS unless the field name ends in `Percent`.
        - `base` number, nullable, required
        - `compare` number, nullable, required
        - `delta` number, nullable, required
        - `percentDelta` number, nullable, required
    - `base` EvalRunCompareCaseSide, required
      - `outcome` 'passed' | 'failed' | 'absent', required
      - `iterationIds` string[], required
      - `representativeIterationId` string, nullable, required
      - `error` string, nullable, required
    - `compare` EvalRunCompareCaseSide, required
      - `outcome` 'passed' | 'failed' | 'absent', required
      - `iterationIds` string[], required
      - `representativeIterationId` string, nullable, required
      - `error` string, nullable, required
    - `metrics` object — Per-case cost, with the coverage that produced it. `costCoverage` is absent when the platform does not report coverage; absence means no opinion, never full coverage, and a cost gate refuses to judge on it.
      - `estimatedCostUsd` NumericDiff, required — A base/compare pair with its delta. Rate-valued instances are FRACTIONS unless the field name ends in `Percent`.
        - `base` number, nullable, required
        - `compare` number, nullable, required
        - `delta` number, nullable, required
        - `percentDelta` number, nullable, required
      - `costCoverage` CostCoverage — How many iterations on each side of a comparison actually contributed a cost. Cost is stamped per iteration and is absent whenever the model was not MCPJam-billed (your own API key), the run used a harness (which mixes models within a turn), or the trial reported no tokens. Summing the priced iterations and skipping the rest yields a figure indistinguishable from a complete one — so a drop in `estimatedCostUsd` is only a cost signal when coverage is full on both sides. A CI gate should treat partial coverage as non-gateable rather than as a pass or a fail.
        - `base` CostCoverageSide, required
          - `costed` integer, required — Iterations that contributed a cost.
          - `total` integer, required — Iterations on this side.
        - `compare` CostCoverageSide, required
          - `costed` integer, required — Iterations that contributed a cost.
          - `total` integer, required — Iterations on this side.

## Other responses

- `401` — Missing, invalid, revoked, or orphaned key (`UNAUTHORIZED`) — or the **target MCP server** needs an OAuth grant (`OAUTH_REQUIRED`), which is a property of the server, not your key.
- `403` — Key is valid but not allowed to do this.
- `404` — Unknown project, server, or resource.
- `429` — Per-key rate limit exceeded (60 requests/minute sustained, bursts up to 10). Honor `Retry-After` and back off with jitter.
- `500` — Something failed on MCPJam's side.
- `502` — Could not connect to the target MCP server.

## Changes

- **2026-09-14** `56b9d1dda7ea` — 1 info
  - added the new optional `header` request parameter `x-mcpjam-eval-vocabulary`
- **2026-09-14** `3ae6331c1446` — 2 warning, 2 info
  - added the new `case` enum value to the `baseRun/modelSource` response property for the response status `200`
  - added the new `case` enum value to the `compareRun/modelSource` response property for the response status `200`
  - added the optional property `baseRun/client` to the response with the `200` status
  - added the optional property `compareRun/client` to the response with the `200` status
- **2026-09-12** `de57cfcdd824` — 1 info
  - added the non-success response with the status `502`
- **2026-09-07** `417b6d81e434` — 2 info
  - added the optional property `cases/items/metrics` to the response with the `200` status
  - added the optional property `metrics/costCoverage` to the response with the `200` status
- **2026-08-27** `6898f438c65e` — 1 info
  - added the optional property `skills` to the response with the `200` status

[Full history](https://skmtc.dev/mcpjam/apis/mcpjam-api/changes/projects/:projectId/eval-runs/:runId/compare/get.md)

---

[API](https://skmtc.dev/mcpjam/apis/mcpjam-api.md) · [All operations](https://skmtc.dev/mcpjam/apis/mcpjam-api/llms.txt) · [OpenAPI document](https://skmtc.dev/mcpjam/apis/mcpjam-api/revisions/56b9d1dda7ea?raw)