Eval runs

Get run status

Changed on

Run status, result, and summary. Poll until status is terminal (completed, failed, cancelled or timed_out). grading is not terminal — every trial has finished and the run is held for its gating judge, with result still pending — so keep polling.

get/projects/{projectId}/eval-runs/{runId}

Request

  • Base URL
    https://app.mcpjam.com/api/v1
  • URL
    https://app.mcpjam.com/api/v1/projects/{projectId}/eval-runs/{runId}
  • Auth
    HTTP bearer

Path parameters

projectIdstring required

ID of the hosted project that contains the server.

runIdstring required

Eval run ID, as returned by POST /eval-runs.

Headers

x-mcpjam-eval-vocabulary'1' | '2'

Which vocabulary this request and its response speak. Absent means 1, which is byte-for-byte today's contract: the same request fields, the same refusals, the same response projection. 2 is the canonical vocabulary. Any other value is a 400 with code: "VALIDATION_ERROR".

Today it decides one thing: the spelling of an evaluator's policy role. Vocabulary 1 accepts and returns gating; vocabulary 2 accepts both spellings and returns the canonical required. Sending required without the header is a 400, deliberately — vocabulary 1 is not widened to meet vocabulary 2 half way, because a boundary that accepts a spelling it does not announce is one two implementations can disagree about.

A response that varies by vocabulary sends Vary: x-mcpjam-eval-vocabulary.

Response

The run.

idstring required
suiteIdstring required
runNumberinteger nullable
status'pending' | 'running' | 'grading' | 'completed' | 'failed' | 'cancelled' | 'timed_out' required

Poll until TERMINAL. The four terminal statuses are completed, failed, cancelled and timed_out. grading is NOT terminal: every trial has finished and the run is being held for its gating judge — up to 30 minutes — with result still pending. A poller that stops at grading reports a run with no verdict as though it had one.

result'passed' | 'failed' | 'inconclusive' | 'null' nullable

Verdict once terminal. inconclusive exists only under verdictPolicyVersion: 2 and is NOT a failure: the run did not measure the server well enough to say (too few gradeable trials, too many evaluator errors), so a gate that folds it into failed reports a defect the run never observed. Read verdictSummary.reasons for the check that withheld the verdict.

source'ui' | 'api' | 'sdk' | 'schedule' | 'github_check' | 'benchmark' required

Run origin, STAMPED BY THE SERVER and not settable by a caller. Every run created through this API is api — including one launched by the CLI, by a GitHub Actions job, or by an MCP agent, because from the server's side all three are API calls. The caller's own claim about which of those it is rides on launcher. schedule and github_check are written by the platform's scheduled-eval and PR-check workers; benchmark marks a Connector Bench matrix cell and is hidden from project run lists.

notesstring nullable
createdAtnumber required

Epoch milliseconds.

completedAtnumber nullable

Epoch milliseconds, null until terminal.

scoreIntegrity'valid' | 'invalid' | 'null' nullable

Whether the run's score evidence verified at ingest. TRI-STATE, and the third state matters: valid means the backend checked and definitions and results agree; invalid means they do not; null (or absent) means NO VERDICT was produced, on a deployment that predates integrity checking. A score gate must treat null exactly like invalid — absent evidence is not valid evidence.

verdictPolicyVersion2

Which criterion decided this run, frozen at run start. 2 is PER-CASE GRADING; ABSENT means the SUITE-WIDE ACCURACY THRESHOLD — result cannot then be inconclusive, because that criterion has no validity phase, and there is no verdictSummary. The 2 is a wire spelling, not a version a caller upgrades to. A caller gating on fractions or on validity must read this FIRST rather than assume a missing summary means a clean run.

verdictSummaryobject

How a policy-2 verdict was reached: the resolved validity policy, the measured completion and evaluator-error rates with their denominators and exclusions, the per-case and per-execution-variant aggregates, and the exact reasons. Absent when the run is legacy, or when the stored summary failed contract validation at the boundary — a partially-valid decision is never published, because a gate cannot tell a missing field from a satisfied check.

verdictPolicyIntegrityErrorstring

Why a policy-2 run could not be decided from its own evidence (a missing or malformed policy snapshot, mixed evaluator configs). Accompanies an inconclusive result; it is never a task failure.

runGroupIdstring

Shared by every per-target run from the same fan-out launch. Absent on a single-target launch and on rows created before run groups.

rerunOfRunIdstring

The run this one re-ran a subset of. Present only with rerunScope.

rerunScope'failed_cases'

failed_cases: this run re-ran only the cases of rerunOfRunId that did not pass. Its pass rate is biased by that selection, so it is never a suite's latest run, a trend point, or a baseline. Absent on every other run.

effectiveModelIdstring

Model the run actually executed with. Absent on pre-attribution rows.

modelSource'client_default' | 'override' | 'case'

client_default inherited the host model; override used the environment's modelId; case used the sole model in the case snapshot.

executionEnginestring

Which engine executed the run: emulated (the platform's own turn loop) or harness:<id> (a real agent runtime such as Claude Code). ABSENT means the run recorded no engine — a run created before the platform attributed one. Treat that as UNKNOWN, never as emulated: those are different claims, and the runs whose engine was never recorded are exactly the ones a reader must not vouch for.

namestring
tagsstring[]
runMetadataobject

Bounded caller metadata; preserved as descriptive data, never authorization.

ciMetadataobject

Bounded CI attribution and source revision metadata.

runEvaluationsByCaseobject[]

Case-scoped run evaluator observations with source and definition identity.

Changes

  • aea5de5036d4Revision changes
    • info

      added the optional property to the response with the status

    • info

      added the optional property to the response with the status

  • ee79c934cffbRevision changes
    • warning

      added the new analysisUnavailable enum value to the /////////// response property for the response status

    • warning

      added the new analysisUnavailable enum value to the ///////////// response property for the response status