--- title: "Get simulation run results" method: GET path: "/v1/simulation/plan/job/{jobId}/results" tags: ["Simulation Run Plan Job"] --- # Get simulation run results `GET /v1/simulation/plan/job/{jobId}/results` ## Request - Base URL: `https://api.roark.ai` - URL: `https://api.roark.ai/v1/simulation/plan/job/{jobId}/results` - Auth: HTTP bearer Everything the run report shows, in one request: the summary and score, the pass/fail verdict, per-metric aggregates, every simulation with its check results, breakdowns by agent and persona, the comparison against the previous run (or `compareToJobId`), the sweep attribution, and the written debrief with its executive summary, failure themes and recommendations. Safe to call while the run is in progress: counts reflect what has landed so far. `verdict` and `sweepAttribution` stay null until the run ends. The debrief is generated automatically a few minutes after a run with graded calls settles; poll until `debrief.status` is READY, or request one with POST /v1/simulation/plan/job/{jobId}/debrief. ## Path parameters - `jobId` string, required ## Query parameters - `compareToJobId` string, uuid: Simulation run plan job to compare this run against, e.g. the last run on your main branch. Defaults to the plan's previous settled run. Any run in the project works, including another plan's: checks are matched by metric, and a check only one run graded reads null on the other side. ## Response `200` The run’s results - object - `data` SimulationRunResults, required: Everything the run report shows, in one response. - `run` SimulationRunResultsRun, required - `simulationRunPlanJobId` string, uuid, required - `simulationRunPlanId` string, uuid, required - `simulationRunPlanName` string, nullable, required: The run plan’s name. Null when the plan has since been deleted. - `runNumber` integer, nullable, required: Per-project run number, shown as SR-42 in the app. - `status` 'PENDING' | 'QUEUED' | 'CREATING_SNAPSHOTS' | 'CREATING_SIMULATIONS' | 'PREPARING_CAPACITY' | 'RUNNING_SIMULATIONS' | 'COMPLETED' | 'FAILED' | 'TIMED_OUT' | 'CANCELLED' | 'CANCELLING' | 'ENDING_SIMULATIONS', required - `triggeredBy` 'SCHEDULED' | 'USER_TRIGGERED_FROM_UI' | 'RE_RUN' | 'TRIGGERED_FROM_API' | 'SYSTEM', required - `createdAt` string, required - `startedAt` string, nullable, required - `endedAt` string, nullable, required - `durationMs` integer, nullable, required: Wall-clock run time. Null until the run ends. - `summary` SimulationRunResultsSummary, required - `totalCalls` integer, required: Every simulation of the run, retries included. - `countedCalls` integer, required: Simulations whose results count: all but the invalidated ones. - `completedCalls` integer, required: Counted simulations that connected and produced a conversation. - `evaluatedCalls` integer, required: Counted simulations at least one check graded. - `passedCalls` integer, required: Evaluated simulations that passed every check. - `failedCalls` integer, required: Evaluated simulations that failed at least one check. - `callPassRate` number, nullable, required: passedCalls / evaluatedCalls, 0-100. Null when nothing was evaluated. - `score` number, nullable, required: The run’s score: the mean of each check’s pass rate, the same number as `verdict.score`. 0-100. Null when nothing was evaluated. - `checkPassRate` number, nullable, required: Every (check, simulation) verdict pooled. 0-100. Null when nothing was evaluated. - `verdict` SimulationRunPlanJobVerdict, required: Pass/fail verdict for the run, judged against the success criteria pinned on the run plan when the run started. - `passed` boolean, required: Whether every check cleared its own `minPassRate` (and the run completed with full coverage). Use this as the CI exit status. Never derived from `score`: a run can score 95 and still fail, or score 40 and still pass. - `score` number, nullable, required: The run's headline quality number, 0-100: the mean of each check's own pass rate. Every check weighs the same, however many simulations it evaluated, which is the same way `passed` treats them. It is the number the Roark dashboard shows for this run. REPORTING ONLY, for dashboards and trend lines. Nothing is judged against it. Null when nothing was evaluated. - `checks` SimulationRunPlanJobCheck[], required: Every check the run was judged on, with its rate and the minimum it had to reach. Thresholds and authored yes/no metrics only. See `SimulationRunPlanJobCheck` for why a provider reading is not one. - `metricDefinitionId` string, uuid, required - `metricName` string, nullable - `passRate` number, nullable, required: This check's own pass rate, 0-100: the share of sims it evaluated that passed it. Null when it evaluated nothing. - `minPassRate` number, required: THE BAR it was judged against: the share of the run's simulations it had to pass. - `inherited` boolean, required: Whether `minPassRate` is the 80% default (`true`) or a minimum this metric set for itself. - `passedSims` integer, required - `evaluatedSims` integer, required - `passed` boolean, required: `passRate >= minPassRate`. - `failures` SimulationRunPlanJobVerdictFailure[], required: Every criterion the run missed. Empty when it passed. - union: One reason the run failed. Every reason is a hard fail: the run passes only when this list is empty. - object - `type` 'RUN_NOT_COMPLETED', required - `status` 'PENDING' | 'QUEUED' | 'CREATING_SNAPSHOTS' | 'CREATING_SIMULATIONS' | 'PREPARING_CAPACITY' | 'RUNNING_SIMULATIONS' | 'COMPLETED' | 'FAILED' | 'TIMED_OUT' | 'CANCELLED' | 'CANCELLING' | 'ENDING_SIMULATIONS', required: The status the run actually ended in. - object - `type` 'INCOMPLETE_COVERAGE', required - `evaluatedCalls` integer, required - `expectedCalls` integer, required - object: Your agent never spoke on more than `maxShare` percent of the run. Those simulations are left out of every check, so the run fails on them whatever the checks say. - `type` 'AGENT_NEVER_SPOKE', required - `neverSpokeCalls` integer, required: Test cases your agent never spoke on, judged on each test case’s last attempt: a silent simulation a retry later reached your agent on does not count here. - `totalCalls` integer, required: Every test case of the run whose last attempt reached your agent. - `maxShare` number, required: The largest share of simulations, 0-100, your agent may never speak on while the run can pass. - `agentName` string, nullable: The agent as the run tested it. Null when the simulations tested more than one agent. - `silentAttempts` integer: Every simulation your agent never spoke on, including the ones a retry replaced. Above `neverSpokeCalls` only when the plan retries silent simulations (`maxNoResponseRetries`). - object - `type` 'METRIC_NOT_EVALUATED', required - `metricDefinitionId` string, uuid, required - `metricName` string, nullable: The check’s name, for rendering the failure. - object - `type` 'METRIC_BELOW_MIN_PASS_RATE', required - `metricDefinitionId` string, uuid, required - `metricName` string, nullable: The check’s name, for rendering the failure. - `passRate` number, required - `minPassRate` number, required - `inherited` boolean, required: Whether the missed minimum was the 80% default (`true`) or this metric's own. - `attemptSummary` SimulationRunPlanJobAttemptSummary, required: The run’s simulations against its test cases. Differs from a plain job count only when the plan retries simulations your agent never spoke on. - `testCaseCount` integer, required: Test cases in the run. Each one has a single final attempt that its result is read from. - `attemptCount` integer, required: Every simulation placed, retries included. Each is a separate, billed call. - `silentAttemptCount` integer, required: Simulations your agent never spoke on, including the ones a retry replaced. - `retryCount` integer, required: Simulations placed as a retry of one your agent never spoke on. - `stillSilentTestCaseCount` integer, required: Test cases whose final attempt your agent still never spoke on. This is what the `AGENT_NEVER_SPOKE` verdict failure counts. - `metrics` SimulationRunMetricResult[], required: Checks first, then other metrics. - `metricDefinitionId` string, uuid, required - `metricId` string, required: The metric’s slug. - `name` string, required - `outputType` 'COUNT' | 'NUMERIC' | 'BOOLEAN' | 'SCALE' | 'TEXT' | 'CLASSIFICATION' | 'OFFSET', required - `isCheck` boolean, required: A threshold: one of the checks the run is judged on. - `unitSymbol` string, nullable, required: Unit of `numeric`, e.g. `ms`. - `evaluatedCalls` integer, required: Counted simulations with at least one value. - `sampleSize` integer, required: Values across those simulations. A per-turn metric has several per call. - `check` object, nullable, required: The check’s tally. Null for a metric that is not a check. - `passedCalls` integer, required - `failedCalls` integer, required - `evaluatedCalls` integer, required - `passRate` number, nullable, required: 0-100. Null when nothing was evaluated. - `numeric` object, nullable, required: For NUMERIC, SCALE, COUNT and OFFSET metrics. Null when nothing was captured, or without the `metric:read` permission. - `average` number, required - `median` number, required - `p95` number, required - `min` number, required - `max` number, required - `distribution` object[], nullable, required: How many values took each reading, for BOOLEAN, SCALE and CLASSIFICATION metrics. Null without the `metric:read` permission. - `value` string, required - `count` integer, required - `calls` SimulationRunCallResult[], required - `simulationJobId` string, uuid, required - `callId` string, uuid, nullable, required: Use with GET /v1/call/{callId} for the transcript, recording and every metric value. - `status` 'QUEUED' | 'CONNECTING' | 'IN_CALL' | 'PROCESSING' | 'COMPLETED' | 'ERROR' | 'TIMED_OUT' | 'CANCELLED' | 'INVALIDATED' | 'NOT_RUN', required: QUEUED, CONNECTING, IN_CALL and PROCESSING while the simulation runs. Once settled: COMPLETED, TIMED_OUT (ran to the maximum duration, still graded), ERROR (failed on the agent’s side), NOT_RUN (failed on Roark’s side), CANCELLED, or INVALIDATED (ran, but does not count). - `counted` boolean, required: False for an invalidated simulation: listed, never scored. - `attemptNumber` integer, required - `retryOfSimulationJobId` string, uuid, nullable, required - `personaName` string, nullable, required - `agentName` string, nullable, required - `durationMs` integer, nullable, required - `passed` boolean, nullable, required: Passed every check that graded it. Null when no check graded it, or it does not count. - `checks` object[], required - `metricDefinitionId` string, uuid, required - `metricName` string, required - `passed` boolean, required - `values` object[], required: Call-level values of the run’s other metrics. Empty without the `metric:read` permission. Per-turn values are on GET /v1/call/{callId}/metrics. - `metricDefinitionId` string, uuid, required - `metricName` string, required - `outputType` 'COUNT' | 'NUMERIC' | 'BOOLEAN' | 'SCALE' | 'TEXT' | 'CLASSIFICATION' | 'OFFSET', required - `participantRole` 'AGENT' | 'CUSTOMER' | 'SIMULATED_CUSTOMER' | 'BACKGROUND_SPEAKER', nullable, required - `value` union, required - number - boolean - string - `reasoning` string, nullable, required - `breakdowns` object, required - `byAgent` SimulationRunResultsGroup[], required - `key` string, required: The agent or persona ID. - `label` string, required - `calls` integer, required: Counted simulations in the group. - `evaluatedCalls` integer, required - `passedCalls` integer, required - `callPassRate` number, nullable, required: 0-100. Null when nothing was evaluated. - `score` number, nullable, required: The group’s own score. 0-100. Null when nothing was evaluated. - `byPersona` SimulationRunResultsGroup[], required - `key` string, required: The agent or persona ID. - `label` string, required - `calls` integer, required: Counted simulations in the group. - `evaluatedCalls` integer, required - `passedCalls` integer, required - `callPassRate` number, nullable, required: 0-100. Null when nothing was evaluated. - `score` number, nullable, required: The group’s own score. 0-100. Null when nothing was evaluated. - `comparison` SimulationRunResultsComparison, required - `baselineRun` object, required - `simulationRunPlanJobId` string, uuid, required - `runNumber` integer, nullable, required - `status` 'PENDING' | 'QUEUED' | 'CREATING_SNAPSHOTS' | 'CREATING_SIMULATIONS' | 'PREPARING_CAPACITY' | 'RUNNING_SIMULATIONS' | 'COMPLETED' | 'FAILED' | 'TIMED_OUT' | 'CANCELLED' | 'CANCELLING' | 'ENDING_SIMULATIONS', required - `endedAt` string, nullable, required - `baselineSource` 'PREVIOUS_RUN' | 'REQUESTED', required: REQUESTED when `compareToJobId` picked the baseline, PREVIOUS_RUN otherwise. - `currentScore` number, nullable, required - `baselineScore` number, nullable, required - `scoreDelta` number, nullable, required: Current minus baseline, in points. - `currentCallPassRate` number, nullable, required - `baselineCallPassRate` number, nullable, required - `callPassRateDelta` number, nullable, required - `checks` object[], required: Every check either run was graded on. A side that did not grade it reads null. - `metricDefinitionId` string, uuid, required - `metricName` string, required - `currentPassRate` number, nullable, required - `baselinePassRate` number, nullable, required - `delta` number, nullable, required - `sweepAttribution` SimulationRunPlanJobSweepAttribution, required: For a run that swept a property (accent, background noise, speech pace and so on): which check failures the property caused. Each value is compared with every other value combined using a one-sided Fisher exact test, and every comparison in the run is corrected together with Benjamini-Hochberg. A value is only called worse when the difference is statistically significant, so a value that happened to fail a few more simulations by chance is not reported as a problem. Invalidated simulations and simulations that failed on the Roark platform are left out. - `values` SimulationRunPlanJobSweepArm[], required: Every value of the swept property, baseline first. - `key` string, required: The arm's identity: the override signature its calls ran with, e.g. `BACKGROUND_NOISE=DRIVING;BACKGROUND_NOISE_VOLUME=0.7`. Join on this rather than on `value`: a sweep can run one value as several arms (Driving at 70% and at 100%), and those share a value. - `value` string, required: The stored value of the swept property, e.g. `US`. - `label` string, required: The value in words, with what else the arm pinned, e.g. `American` or `Driving (70% noise)`. - `isBaseline` boolean, required: Whether this is the baseline the plan named. - `attempted` integer, required: Simulations run at this value. Simulations that failed on the Roark platform are left out, since they say nothing about your agent. - `counted` integer, required: Simulations that actually tested the property: not invalidated, and graded by at least one check. Only these are used in the comparison. - `silentAttempts` integer, required: Of `attempted`, the simulations your agent never spoke on, whether or not a retry followed. They are invalidated, so they are never in `counted`. - `retries` integer, required: Of `attempted`, the retries of simulations your agent never spoke on. Zero unless the plan retries silent simulations. - `testable` boolean, required: Whether this value had at least `minArmCalls` counted simulations. A value below that is "insufficient data": it is reported with its numbers but never compared or ranked, because a rate from one or two calls cannot be told apart from luck. - `score` number, nullable, required: The mean of each check's pass rate at this value, 0-100, the same rule as the run's `score`. Descriptive only: a lower score alone never makes a value worse. Null when nothing counted. - `checks` SimulationRunPlanJobSweepCheck[], required: Every check the run was graded on: `PROPERTY_ATTRIBUTABLE` first, then `FOUND_UNATTRIBUTED`, then `WITHIN_NOISE`. - `metricDefinitionId` string, uuid, required - `metricName` string, required - `attribution` 'PROPERTY_ATTRIBUTABLE' | 'FOUND_UNATTRIBUTED' | 'WITHIN_NOISE', required: How this check relates to the swept property: - `PROPERTY_ATTRIBUTABLE`: at least one value failed it significantly more often than every other value combined. `worseValues` names them. This is the only case in which a value is called worse. - `FOUND_UNATTRIBUTED`: no value stands out, but the check failed on at least `foundUnattributedFailureRate`% of counted simulations overall. A real issue with the agent on which no value stood out, so the run cannot tie it to the property. It does not show the property had no effect: a small run may be unable to see one. - `WITHIN_NOISE`: neither. Any differences between values are within what chance produces. - `failed` integer, required - `evaluated` integer, required - `failureRate` number, nullable, required: Share of counted simulations across every value that failed the check, 0-100. - `worseValues` SimulationRunPlanJobSweepArmCheck[], required: The values significantly worse than the rest, most significant first. Empty unless `attribution` is `PROPERTY_ATTRIBUTABLE`. - `key` string, required: The arm's identity: the override signature its calls ran with, e.g. `BACKGROUND_NOISE=DRIVING;BACKGROUND_NOISE_VOLUME=0.7`. Join on this rather than on `value`: a sweep can run one value as several arms (Driving at 70% and at 100%), and those share a value. - `value` string, required - `label` string, required - `failed` integer, required - `evaluated` integer, required - `failureRate` number, nullable, required: Share of the counted simulations at this value that failed the check, 0-100. - `restFailureRate` number, nullable, required: The same share across every other value combined, 0-100. This is what the value is compared with. - `adjustedPValue` number, required: How likely a difference at least this large would be by chance alone, after correcting for every comparison in the run (Benjamini-Hochberg). The value is called worse only when this is at most `falseDiscoveryRate`. - `testableValueCount` integer, required: How many values had enough counted simulations to be compared. - `testedComparisonCount` integer, required: How many value and check pairs were actually tested. A value can have enough simulations and still go untested (no other value graded that check, or too few verdicts on it). When this is 0 no comparison ran, so an empty `worseValues` everywhere means nothing was tested, not that no value did worse. - `minimumDetectableGap` number, nullable, required: Roughly how many percentage points more often a value would need to fail than the rest of the run to be flagged at this sample size. Optimistic: it uses simulation counts rather than the verdicts on each check and ignores the correction, so checks graded on fewer simulations need larger gaps. Large when few simulations ran per value: finding no significant difference then means the run could not see one, not that none exists. - `caveats` string[], required: Plain sentences on what this run could not see, ready to show to a reader. - `neverSpokeValues` SimulationRunPlanJobSweepNeverSpokeValue[], required: The values your agent never spoke on significantly more often than the rest of the run, most significant first, tested the same way as `worseValues`. Empty on a run with no silent simulations. - `key` string, required: The arm's identity: the override signature its calls ran with, e.g. `BACKGROUND_NOISE=DRIVING;BACKGROUND_NOISE_VOLUME=0.7`. Join on this rather than on `value`: a sweep can run one value as several arms (Driving at 70% and at 100%), and those share a value. - `value` string, required - `label` string, required - `silentAttempts` integer, required: Simulations at this value your agent never spoke on, retried or not. - `attempted` integer, required: Simulations run at this value. - `retries` integer, required: Retries placed at this value because your agent never spoke. - `restSilentAttempts` integer, required: Simulations your agent never spoke on across every other value combined. - `restAttempted` integer, required: Simulations run across every other value combined. - `adjustedPValue` number, required: How likely a difference at least this large would be by chance alone, after correcting across the values (Benjamini-Hochberg). - `minArmCalls` integer, required: Counted simulations a value needs before it is compared. - `falseDiscoveryRate` number, required: The Benjamini-Hochberg false discovery rate the comparisons are held to. - `foundUnattributedFailureRate` number, required: The overall failure rate, 0-100, at which a check with no standout value is reported as `FOUND_UNATTRIBUTED`. - `debrief` SimulationRunDebrief, required: The run’s written debrief: executive summary, failure themes and recommendations. Generated automatically when a run with graded calls settles. - `debriefId` string, uuid, required - `status` 'PENDING' | 'GENERATING' | 'READY' | 'FAILED', required: PENDING and GENERATING while it is written (usually a few minutes), then READY or FAILED. - `errorMessage` string, nullable, required - `createdAt` string, required - `completedAt` string, nullable, required - `report` SimulationRunDebriefReport, required - `generatedAt` string, required - `verdict` 'PASSED' | 'THRESHOLD_FAILED', required - `verdictSummary` string, required: One-paragraph verdict, e.g. "Not ready to promote. ..." - `executiveSummary` object, required - `paragraphs` string[], required - `sections` object[], required - `heading` string, nullable, required - `body` string, required - `keyFindings` object[], required - `title` string, required - `body` string, required - `callIds` string[], required: The calls the finding is about. - `wentWell` object[], required - `title` string, required - `body` string, required - `callIds` string[], required: The calls the finding is about. - `themes` object[], required: Failure themes, clustered across the run’s failing calls. - `slug` string, required - `title` string, required - `severity` 'HIGH' | 'MEDIUM' | 'LOW', required - `impact` 'CRITICAL' | 'HIGH' | 'MEDIUM' | 'LOW', nullable, required - `prevalence` 'WIDESPREAD' | 'RECURRING' | 'ISOLATED', nullable, required - `lifecycle` 'NEW' | 'PERSISTENT' | 'IMPROVING' | 'FIXED', nullable, required - `affectedCallCount` integer, required - `totalCallCount` integer, required - `checkNames` string[], required - `rootCauseChannel` 'PROMPT' | 'VOICE_CONFIG' | 'STT' | 'SPEECH_MODEL' | 'LLM_MODEL' | 'INFRA' | 'EXTERNAL', nullable, required - `analysis` string, required - `callIds` string[], required - `recommendations` object[], required - `order` integer, required - `title` string, required - `body` string, required - `severity` 'HIGH' | 'MEDIUM' | 'LOW', nullable, required - `channel` 'PROMPT' | 'VOICE_CONFIG' | 'STT' | 'SPEECH_MODEL' | 'LLM_MODEL' | 'INFRA' | 'EXTERNAL', nullable, required - `currentText` string, nullable, required: For a prompt fix: the text to replace. - `suggestedText` string, nullable, required: For a prompt fix: the replacement. - `themeSlugs` string[], required - `trend` object, nullable, required: The checks against the previous run, as of generation. - `baselineRunPlanJobId` string, required - `baselineRunNumber` integer, nullable, required - `rows` object[], required - `checkName` string, required - `previousPassRate` number, nullable, required - `currentPassRate` number, nullable, required - `lifecycle` 'NEW' | 'PERSISTENT' | 'IMPROVING' | 'FIXED', nullable, required ## Other responses - `400`: Bad Request - `401`: Unauthorized - `403`: Forbidden - `404`: Simulation run, or the `compareToJobId` run, not found - `429`: Too Many Requests - `500`: Internal Server Error ## Changes > 188 revisions in range; 1 not diffed. - **2026-10-11** `776a680ac9a5`: 1 info - endpoint added [Operation history](https://skmtc.dev/roarkhq/apis/roark-analytics-api/changes/v1/simulation/plan/job/:jobId/results/get.md) --- [API](https://skmtc.dev/roarkhq/apis/roark-analytics-api.md) - [All operations](https://skmtc.dev/roarkhq/apis/roark-analytics-api/llms.txt) - [OpenAPI document](https://skmtc.dev/roarkhq/apis/roark-analytics-api/revisions/efc3377249d1?raw)