---
title: "Compare a run against a baseline"
method: GET
path: "/projects/{projectId}/eval-runs/{runId}/compare"
tags: ["Eval runs"]
---

# Compare a run against a baseline

`GET /projects/{projectId}/eval-runs/{runId}/compare`

Compare this run against a baseline run: per-case status (`regressed`, `fixed`, `new_case`, `removed_case`, `changed`), per-scorer pass-rate and mean deltas from the evaluation contract, and whether the evaluation config changed. Omit `baseRunId` to compare against the nearest earlier **completed** run in the same suite. Returns `404` with `details.reason` = `BASELINE_NOT_FOUND` when there is no comparable predecessor — that is an incomplete comparison, not a failing one. A scorer whose `definitionChanged` is `true` was graded by a different definition on each side, so its delta is not a regression.

## Path parameters

- `projectId` string, required
- `runId` string, required

## Query parameters

- `baseRunId` string
- `baseCommitSha` string

## Response `200`

The comparison.

- EvalRunCompare
  - `suite` object, required
    - `id` string, required
    - `name` string, required
  - `baseline` object, required
    - `policy` 'previous_completed' | 'previous_completed_same_environment' | 'run' | 'commit_sha', required — How the baseline was chosen. `commit_sha` means it was resolved from the requested source SHA.
    - `baseRunId` string, required
    - `baseCommitSha` string — The pinned source SHA, echoed back for the `commit_sha` policy only.
    - `matchCount` integer — Present ONLY when uniqueness could not be established - the SHA matched several eligible runs, or the bounded lookup saturated. Absent means the match was unambiguous; do not default it to 1.
    - `matchCountTruncated` boolean — `matchCount` is a FLOOR, not a total - including when it reads 1. Read it together with `matchCount`; a count without this flag would assert a uniqueness nobody checked.
  - `baseRun` EvalRunCompareSide, required
    - `id` string, required
    - `runNumber` integer, required
    - `result` string, required
    - `createdAt` integer, required
    - `completedAt` integer, nullable, required
    - `summary` object, nullable, required
      - `total` integer, required
      - `passed` integer, required
      - `failed` integer, required
      - `passRate` number, required
    - `environment` object
      - `id` string, required
      - `name` string, nullable, required
    - `effectiveModelId` string — Model the compared run actually executed with.
    - `modelSource` 'client_default' | 'override'
  - `compareRun` EvalRunCompareSide, required
    - `id` string, required
    - `runNumber` integer, required
    - `result` string, required
    - `createdAt` integer, required
    - `completedAt` integer, nullable, required
    - `summary` object, nullable, required
      - `total` integer, required
      - `passed` integer, required
      - `failed` integer, required
      - `passRate` number, required
    - `environment` object
      - `id` string, required
      - `name` string, nullable, required
    - `effectiveModelId` string — Model the compared run actually executed with.
    - `modelSource` 'client_default' | 'override'
  - `passSummary` object, required — Run-summary counters. Named `passSummary`, not `scores`, so it cannot be confused with `scoreContract` — the two answer different questions.
    - `passRatePercent` NumericDiff, required — A base/compare pair with its delta. Rate-valued instances are FRACTIONS unless the field name ends in `Percent`.
      - `base` number, nullable, required
      - `compare` number, nullable, required
      - `delta` number, nullable, required
      - `percentDelta` number, nullable, required
    - `total` NumericDiff, required — A base/compare pair with its delta. Rate-valued instances are FRACTIONS unless the field name ends in `Percent`.
      - `base` number, nullable, required
      - `compare` number, nullable, required
      - `delta` number, nullable, required
      - `percentDelta` number, nullable, required
    - `passed` NumericDiff, required — A base/compare pair with its delta. Rate-valued instances are FRACTIONS unless the field name ends in `Percent`.
      - `base` number, nullable, required
      - `compare` number, nullable, required
      - `delta` number, nullable, required
      - `percentDelta` number, nullable, required
    - `failed` NumericDiff, required — A base/compare pair with its delta. Rate-valued instances are FRACTIONS unless the field name ends in `Percent`.
      - `base` number, nullable, required
      - `compare` number, nullable, required
      - `delta` number, nullable, required
      - `percentDelta` number, nullable, required
  - `metrics` object, required
    - `wallDurationMs` NumericDiff, required — A base/compare pair with its delta. Rate-valued instances are FRACTIONS unless the field name ends in `Percent`.
      - `base` number, nullable, required
      - `compare` number, nullable, required
      - `delta` number, nullable, required
      - `percentDelta` number, nullable, required
    - `totalTokens` NumericDiff, required — A base/compare pair with its delta. Rate-valued instances are FRACTIONS unless the field name ends in `Percent`.
      - `base` number, nullable, required
      - `compare` number, nullable, required
      - `delta` number, nullable, required
      - `percentDelta` number, nullable, required
    - `estimatedCostUsd` NumericDiff, required — A base/compare pair with its delta. Rate-valued instances are FRACTIONS unless the field name ends in `Percent`.
      - `base` number, nullable, required
      - `compare` number, nullable, required
      - `delta` number, nullable, required
      - `percentDelta` number, nullable, required
  - `skills` object, nullable — Which skills changed between the two runs — the configuration attribution that usually explains the case-level differences. `null` when NEITHER run recorded pinned skills (an empty section would instead claim no skills were involved). Absent when the deployment predates skill attribution, so clients must tolerate all three states.
    - `base` EvalRunCompareSkillsSide, required
      - `excluded` boolean, required — This run deliberately ran with skills disabled — distinct from a run that simply pinned none.
      - `count` integer, required — How many skills this run pinned.
    - `compare` EvalRunCompareSkillsSide, required
      - `excluded` boolean, required — This run deliberately ran with skills disabled — distinct from a run that simply pinned none.
      - `count` integer, required — How many skills this run pinned.
    - `changes` EvalRunCompareSkillChange[], required — Added, removed and changed skills only, changed first. Unchanged skills are counted, not listed.
      - `key` string, required — Stable match key; opaque, safe for list keys and dedupe.
      - `name` string, required
      - `modelRef` string — Namespaced runtime address for a plugin-channel skill.
      - `channels` string[], required
      - `kind` 'added' | 'removed' | 'changed', required
      - `renamedFrom` string — Present when the skill was renamed between the runs; it is still matched as ONE skill by its logical id.
      - `base` EvalRunCompareSkillSide — One skill's identity on one side of the comparison.
        - `contentHash` string, required
        - `aggregateHash` string — Complete-artifact hash; present only when supporting files diverge it from `contentHash`.
        - `versionNumber` integer — Authored-skill revision, when the run recorded one.
        - `serverSkillVersionNumber` integer — MCP-captured revision, when the run recorded one.
      - `compare` EvalRunCompareSkillSide — One skill's identity on one side of the comparison.
        - `contentHash` string, required
        - `aggregateHash` string — Complete-artifact hash; present only when supporting files diverge it from `contentHash`.
        - `versionNumber` integer — Authored-skill revision, when the run recorded one.
        - `serverSkillVersionNumber` integer — MCP-captured revision, when the run recorded one.
      - `versionDelta` string — Human-readable revision move (`v3 → v4`), present only when BOTH sides recorded a number. A change with no delta is a real content change whose revisions are unknown, not a smaller change.
    - `unchangedCount` integer, required
  - `scoreContract` object, required
    - `base` ScoreContractSide, required
      - `evaluationConfigHash` string, nullable, required
      - `scoreIntegrity` 'valid' | 'invalid' | 'null', nullable, required — `null` means no verdict was produced. A gate must treat it exactly like `invalid` — absent evidence is not valid evidence.
      - `scoredIterations` integer, required
      - `quarantinedIterations` integer, required — Iterations with at least one row that failed to verify at ingest. Counted here, and excluded from every rate.
    - `compare` ScoreContractSide, required
      - `evaluationConfigHash` string, nullable, required
      - `scoreIntegrity` 'valid' | 'invalid' | 'null', nullable, required — `null` means no verdict was produced. A gate must treat it exactly like `invalid` — absent evidence is not valid evidence.
      - `scoredIterations` integer, required
      - `quarantinedIterations` integer, required — Iterations with at least one row that failed to verify at ingest. Counted here, and excluded from every rate.
    - `evaluationConfigChanged` boolean, required
    - `scorers` ScoreContractScorer[], required
      - `scorerId` string, required
      - `gating` boolean, required
      - `deterministic` boolean, required
      - `definitionChanged` boolean, required — The same scorer id was graded under a different definition hash on each side. Its delta is NOT a regression — the two runs did not measure the same thing.
      - `passRate` NumericDiff, required — A base/compare pair with its delta. Rate-valued instances are FRACTIONS unless the field name ends in `Percent`.
        - `base` number, nullable, required
        - `compare` number, nullable, required
        - `delta` number, nullable, required
        - `percentDelta` number, nullable, required
      - `meanValue` NumericDiff, required — A base/compare pair with its delta. Rate-valued instances are FRACTIONS unless the field name ends in `Percent`.
        - `base` number, nullable, required
        - `compare` number, nullable, required
        - `delta` number, nullable, required
        - `percentDelta` number, nullable, required
      - `errorCount` object, required
        - `base` integer, required
        - `compare` integer, required
  - `cases` EvalRunCompareCase[], required
    - `caseKey` string, required
    - `title` string, required
    - `status` 'unchanged_passed' | 'unchanged_failed' | 'regressed' | 'fixed' | 'new_case' | 'removed_case' | 'changed', required
    - `configChanged` boolean, required — The scenario's own config (prompt, steps, expectations) changed.
    - `evaluationConfigChanged` boolean, required — This case's evaluation config changed.
    - `scoreDeltas` CaseScoreDelta[], required
      - `scorerId` string, required
      - `gating` boolean, required
      - `deterministic` boolean, required
      - `definitionChanged` boolean, required
      - `base` CaseScoreSide, required
        - `status` 'scored' | 'error' | 'skipped' | 'not_applicable', required
        - `value` number, nullable, required
        - `passed` boolean, nullable, required
      - `compare` CaseScoreSide, required
        - `status` 'scored' | 'error' | 'skipped' | 'not_applicable', required
        - `value` number, nullable, required
        - `passed` boolean, nullable, required
      - `value` NumericDiff, required — A base/compare pair with its delta. Rate-valued instances are FRACTIONS unless the field name ends in `Percent`.
        - `base` number, nullable, required
        - `compare` number, nullable, required
        - `delta` number, nullable, required
        - `percentDelta` number, nullable, required
    - `base` EvalRunCompareCaseSide, required
      - `outcome` 'passed' | 'failed' | 'absent', required
      - `iterationIds` string[], required
      - `representativeIterationId` string, nullable, required
      - `error` string, nullable, required
    - `compare` EvalRunCompareCaseSide, required
      - `outcome` 'passed' | 'failed' | 'absent', required
      - `iterationIds` string[], required
      - `representativeIterationId` string, nullable, required
      - `error` string, nullable, required

## Other responses

- `401` — Missing, invalid, revoked, or orphaned key (`UNAUTHORIZED`) — or the **target MCP server** needs an OAuth grant (`OAUTH_REQUIRED`), which is a property of the server, not your key.
- `403` — Key is valid but not allowed to do this.
- `404` — Unknown project, server, or resource.
- `429` — Per-key rate limit exceeded (60 requests/minute sustained, bursts up to 10). Honor `Retry-After` and back off with jitter.
- `500` — Something failed on MCPJam's side.

## Changes

- **2026-08-27** `6898f438c65e` — 1 info
  - added the optional property `skills` to the response with the `200` status
- **2026-08-25** `31c2408136d8` — 1 warning, 4 info
  - added the new `commit_sha` enum value to the `baseline/policy` response property for the response status `200`
  - added the new optional `query` request parameter `baseCommitSha`
  - added the optional property `baseline/baseCommitSha` to the response with the `200` status
  - added the optional property `baseline/matchCount` to the response with the `200` status
  - …1 more
- **2026-08-22** `de73fc9b1118` — 1 warning, 6 info
  - added the new `previous_completed_same_environment` enum value to the `baseline/policy` response property for the response status `200`
  - added the optional property `baseRun/effectiveModelId` to the response with the `200` status
  - added the optional property `baseRun/environment` to the response with the `200` status
  - added the optional property `baseRun/modelSource` to the response with the `200` status
  - …3 more
- **2026-08-15** `7d1aff84ce3b` — 1 info
  - endpoint added

[Change history](https://skmtc.dev/mcpjam/apis/mcpjam-api/changes/projects/:projectId/eval-runs/:runId/compare/get.md)

---

[API](https://skmtc.dev/mcpjam/apis/mcpjam-api.md) · [All operations](https://skmtc.dev/mcpjam/apis/mcpjam-api/llms.txt) · [OpenAPI document](https://skmtc-service-production.skmtc.workers.dev/v1/apis/mcpjam/mcpjam-api/revisions/c628a0e95b06/schema)