---
title: "Run a test suite and wait for the result"
method: POST
path: "/api/cli/run"
tags: ["CI & Test Runs"]
---

# Run a test suite and wait for the result

`POST /api/cli/run`

## Request

- Base URL: `https://api.routebase.dev`
- URL: `https://api.routebase.dev/api/cli/run`
- Auth: API key in header `X-API-Key`

Executes a test suite and returns the outcome in one call, which is what a
pipeline step needs. The response tells you whether everything passed, so a
CI job can fail on `allPassed` alone. Pass a `reportFormat` to get the report
inline instead of downloading it separately.

The call blocks until the run finishes. For a long suite, send `progressRunId`
and poll the status endpoint instead of holding the connection open.

## Headers

- `X-RB-Region` string

## Request body

- CliRunRequest: Everything a pipeline needs to start one suite run.
  - `projectId` string, uuid, required: Public id of the project.
  - `suiteId` string, uuid, required: Public id of the test suite to run.
  - `environmentId` string, uuid, nullable: Environment whose variables the run substitutes.
  - `iterations` integer, nullable: How often to run the suite, between 1 and 10000. Defaults to 1.
  - `delayMs` integer, nullable: Pause between iterations in milliseconds, up to 60000. Defaults to 0.
  - `onError` 'ignore' | 'stop' | 'abort': What a run does when a test case errors. `ignore` continues, `stop` ends the current iteration but starts the next one, and `abort` ends the whole run.
  - `reportFormat` 'json' | 'html' | 'junit', nullable: Include the report inline in the response in this format. Omit it to get no report.
  - `threads` integer, nullable: Parallel threads for performance mode, between 1 and 50. Defaults to 1, which runs the cases in order and carries variables between them.
  - `authOverrideJson` string, nullable: Authentication configuration that replaces the one stored on the suite, which lets a pipeline inject its own credentials.

## Response `200`

The run completed. Check `allPassed` for the verdict.

- CliRunResponse: The verdict of a suite run, shaped for a CI step. Branch your build on allPassed.
  - `allPassed` boolean, required: Whether every test case passed across every iteration. Gate your build on this.
  - `totalElapsedMs` integer, required: Total run time in milliseconds.
  - `totalTests` integer, required: Number of test case executions.
  - `passedTests` integer, required: Number that passed.
  - `failedTests` integer, required: Number that failed or errored.
  - `testRunId` string, uuid, nullable: Public id of the stored test run, which the report endpoints take.
  - `iterations` integer, required: Number of iterations executed.
  - `iterationResults` IterationResult[], nullable: Per-iteration breakdown, present only when more than one iteration ran.
    - `iterationNumber` integer, required: The 1-based iteration number.
    - `results` TestCaseExecutionResult[], required: The result of every test case in this iteration, in run order.
      - `statusCode` integer, required: Status code from the target server, or 0 when the request never went out.
      - `statusText` string, nullable: Reason phrase from the target server.
      - `headers` object, required: Response headers from the target server.
      - `body` string, nullable: Response body, which may be null for an empty response.
      - `elapsedMilliseconds` integer, required: Request duration in milliseconds.
      - `error` string, nullable: Set when the request itself failed, for example on a timeout.
      - `resolvedUrl` string, required: The URL after variable substitution.
      - `resolvedMethod` string, required: The HTTP method used.
      - `unresolvedVariables` string[], nullable: Variables that could not be resolved, which is usually a missing environment value.
      - `passed` boolean, nullable: Whether every assertion held. Null when the case has no assertions or never ran.
      - `assertionResults` AssertionResult[], nullable: One entry per assertion on the case. Null when the case carries none or never ran.
        - `assertionId` string, uuid, required: Public id of the assertion, stable across runs.
        - `type` string, required: What the assertion checks, for example the status code or a JSON path.
        - `operator` string, required: How expected and actual are compared.
        - `target` string, nullable: What the assertion points at, for example a JSON path.
        - `expectedValue` string, nullable: The value the assertion demanded, rendered as text.
        - `actualValue` string, nullable: The value the response actually carried, rendered as text.
        - `passed` boolean, required: Whether this single assertion held.
        - `errorMessage` string, nullable: Why the assertion failed.
        - `checkedVersionNumber` string, nullable: Specification version the contract assertion checked against.
        - `checkedVersionSource` string, nullable: Where that version came from.
        - `driftedFromVersionNumber` string, nullable: Set when the response no longer matches the version it used to.
      - `testCaseId` string, uuid, nullable: Public id of the test case.
      - `testCaseName` string, nullable: Name of the test case.
      - `extractedVariables` object, nullable: Variables this response contributed to later cases.
      - `consumedVariables` object, nullable: Variables this case read.
      - `resolvedHeaders` object, nullable: Request headers after variable substitution.
      - `resolvedBody` string, nullable: Request body after variable substitution.
      - `responseSizeBytes` integer, nullable: Size of the response body.
      - `preRequestScriptOutput` string[], nullable: Console output of the pre-request script.
      - `preRequestScriptError` string, nullable: Set when the pre-request script failed.
      - `postResponseScriptOutput` string[], nullable: Console output of the post-response script.
      - `postResponseScriptError` string, nullable: Set when the post-response script failed.
      - `timingBreakdown` TimingBreakdown: Where the time inside one request went, from name resolution to the last byte.
        - `dnsMs` integer, required: Time spent resolving the hostname. Zero on a reused connection.
        - `connectMs` integer, required: Time spent opening the TCP connection.
        - `tlsMs` integer, required: Time spent on the TLS handshake.
        - `firstByteMs` integer, required: Time from request sent to the first response byte, which is where server work shows up.
        - `downloadMs` integer, required: Time spent reading the response body.
        - `totalMs` integer, required: Total wall time of the request. It covers the phases above plus scheduling overhead.
      - `isPending` boolean, required: The case has not run yet.
      - `isBlocked` boolean, required: The case was never sent, because it writes and the environment is read-only. This is neither a pass nor a failure, so do not fail a build on it alone.
    - `allPassed` boolean, required: Whether every case passed in this iteration.
    - `elapsedMs` integer, required: Duration of this iteration.
    - `wasAborted` boolean, required: Whether the iteration stopped early because of the error behaviour.
    - `dataRowIndex` integer, nullable: The 0-based data row, for a data-driven run.
    - `dataRowValues` object, nullable: The data row values used, for a data-driven run.
  - `report` string, nullable: The report content, present only when `reportFormat` was requested.
  - `reportFormat` string, nullable: Format of the inline report.
  - `reportContentType` string, nullable: MIME type of the inline report.

## Other responses

- `400`: The request was malformed or failed validation.
- `401`: The API key is missing, invalid, expired or revoked. A US organization calling without `X-RB-Region: us` also lands here, because the request reached the wrong region.
- `403`: The key is valid but lacks the permission or the project scope for this call. A scoped key is also refused on organization level operations by design.
- `404`: The resource does not exist, or it belongs to another organization or project. Both cases answer the same way on purpose, so the API cannot be used to probe for foreign identifiers.
- `429`: The rate limit was exceeded. The `Retry-After` header gives the seconds to wait.

---

[API](https://skmtc.dev/routebase/apis/routebase-public-api.md) · [All operations](https://skmtc.dev/routebase/apis/routebase-public-api/llms.txt) · [OpenAPI document](https://skmtc.dev/routebase/apis/routebase-public-api/revisions/7187df489f27?raw)