---
title: "Run test cases from a JSON spec file"
method: POST
path: "/test_framework/v2/scenarios/run_scenarios_json/"
tags: ["Evaluators"]
---

# Run test cases from a JSON spec file

`POST /test_framework/v2/scenarios/run_scenarios_json/`

## Request

- Base URL: `https://api.cekura.ai`
- URL: `https://api.cekura.ai/test_framework/v2/scenarios/run_scenarios_json/`
- Auth: one of:
  - API key in header `X-CEKURA-API-KEY`
  - HTTP bearer
  - HTTP bearer

Run agent test cases straight from a JSON spec, without creating evaluators. Keep the spec in your repository and post it from CI.

The spec describes only what to test. Pick the target with `agent_id` and `channel` on the request — `voice` for a phone call (default), `text` for chat, `elevenlabs`, `livekit_v2`, or `pipecat_v2` for the matching WebRTC connection, or `chirp` for a raw-PCM WebSocket endpoint — so one file runs unchanged against any agent or environment. LiveKit, Pipecat and CHIRP connection overrides belong in the request's `livekit_data`, `pipecat_data` or `chirp_data` (`websocket_url`), never in the spec.

**Run overrides:** the request accepts the same run-level overrides as `scenarios_run_voice`, applied to every test case in the spec — `frequency` (multiplies each case's own `frequency`), `agent_number` and `outbound_phone_number` (`voice` and `text`; on `voice`, `outbound_phone_number` is for inbound agents only), `mode` (inbound `voice` agents), `concurrency_limit` (overrides the spec's `defaults.concurrency_limit`) and `mock_tool_names`. Personalities, test profiles and the Cekura-side `phone_number` are set per test case in the spec.

Pass `dry_run=true` to validate only: the response describes the run that would happen (test cases, total runs, estimated cost) and nothing is created or charged. Use that on every commit, and the real call on the branches you want tested.

Every problem in a spec is reported at once, keyed by its location in the file (for example `scenarios[3].metrics[1]`). Metrics must already exist and be enabled for the target agent; personalities and test profiles can be referenced by ID or defined inline.

Fetch the schema this endpoint accepts from `scenarios_json_schema`.

**Cost:** voice- or text-simulation rate per run. Test cases from a spec are removed automatically once no run needs them.

## Query parameters

- `dry_run` boolean

## Request body

- TestSuiteSpecUpload: Accepts a spec as a JSON body or an uploaded file. CI posts JSON directly; the dashboard uploads the same file the customer has in their repo. Both land in ``spec``.
  - `spec` unknown
  - `file` string, uri: A .json spec file. Use this instead of 'spec' for uploads.
  - `agent_id` integer: Agent to test. Overrides 'agent_id' in the spec, letting one spec file run against several environments.
  - `name` string: Label for this run. Defaults to the spec's suite name.
  - `channel` 'voice' | 'text' | 'elevenlabs' | 'livekit_v2' | 'pipecat_v2' | 'chirp': How to reach the agent: 'voice' for a phone call (the default), 'text' for chat, 'elevenlabs' for an ElevenLabs WebRTC session, 'livekit_v2' or 'pipecat_v2' for their respective WebRTC sessions, or 'chirp' for the agent's raw-PCM WebSocket endpoint (CHIRP protocol). * `voice` - voice * `text` - text * `elevenlabs` - elevenlabs * `livekit_v2` - livekit_v2 * `pipecat_v2` - pipecat_v2 * `chirp` - chirp
  - `frequency` integer: Run every test case this many times, on top of each case's own 'frequency' in the spec. Default 1.
  - `agent_number` string: Voice and text only. The agent's phone number for this run, in E.164 format, overriding the agent's configured contact number.
  - `outbound_phone_number` string: Voice and text only. For an inbound voice agent, the number Cekura dials, in E.164 format; defaults to 'agent_number', then the agent's contact number. Rejected for outbound voice agents: set 'phone_number' per test case in the spec instead. For SMS and WhatsApp agents, the contact number for the conversation.
  - `mode` 'same_number' | 'different_numbers': Voice, inbound agents only. 'same_number' (default) places every call from the same Cekura number; 'different_numbers' spreads them across the pool and cannot be combined with 'phone_number' in the spec. * `same_number` - same_number * `different_numbers` - different_numbers
  - `concurrency_limit` integer: Cap on the number of runs executed in parallel. Overrides the spec's 'defaults.concurrency_limit'.
  - `calls_per_second` number, double: Most calls started per second for this run, a number from 0.1 to 4 (e.g. 0.5 starts one call every 2 seconds). Overrides the spec's 'defaults.calls_per_second'. Leave both unset for no limit.
  - `mock_tool_names` string[], nullable: Names of mock tools to activate for this run. Omit (or send null) to mock every tool configured on the agent; send an empty list to mock none.
  - `livekit_data` object: Optional LiveKit overrides. These apply only to this run and are not part of the portable test-suite spec.
  - `pipecat_data` object: Optional Pipecat Cloud overrides. These apply only to this run and are not part of the portable test-suite spec.
  - `chirp_data` object: Optional CHIRP overrides. 'websocket_url' replaces the agent's stored raw-PCM WebSocket endpoint for this run only — for example a tunnel URL that changes on every CI run. Not part of the portable test-suite spec.

## Response `200`

- object
  - `id` integer: ID of the result
  - `agent` integer: ID of the agent
  - `status` 'running' | 'completed' | 'failed' | 'pending' | 'in_progress' | 'evaluating' | 'in_queue' | 'timeout' | 'cancelled' | 'scaling_up': Status of the result. Poll until it is `completed`, `failed`, `timeout` or `cancelled`. It stays non-terminal until every run has ended, then becomes `completed` if any run completed — even if runs failed their checks or others errored.
  - `run_as_text` boolean: Whether the scenario ran as text or not
  - `runs` object[]
    - `id` integer: ID of the run
    - `status` 'running' | 'completed' | 'failed' | 'pending' | 'in_progress' | 'evaluating' | 'in_queue' | 'timeout' | 'cancelled' | 'scaling_up': Status of the run
    - `scenario` integer: ID of the scenario
    - `number` string: For outbound runs (agent.inbound=False). The given number that must be called from the phone number configured in the cekura agent.
    - `inbound_number` string, nullable: For inbound runs (agent.inbound=True). The agent's configured phone number will receive calls from this number.
    - `scenario_name` string: Name of the scenario
    - `test_profile_data` object, nullable: Details of the test profile associated with this run scenario
    - `outbound_dial_window_opens_at` string, date-time, nullable: When the outbound dial window opened (run transitioned to pending/waiting-for-call). Null until dispatched.
    - `outbound_dial_window_seconds` integer, nullable: Duration of the outbound dial window in seconds.
    - `outbound_dial_window_closes_at` string, date-time, nullable: Deadline by which your agent must dial the outbound number (opens_at + outbound_dial_window_seconds). Dials after this are dropped and the run returns status: timeout.
  - `created_at` string, date-time

## Other responses

- `400`

## Changes

- **2026-11-03** `a6fb6a739212`: 1 info
  - endpoint added

[Operation history](https://skmtc.dev/cekura/apis/cekura-api/changes/test_framework/v2/scenarios/run_scenarios_json/post.md)

---

[API](https://skmtc.dev/cekura/apis/cekura-api.md) - [All operations](https://skmtc.dev/cekura/apis/cekura-api/llms.txt) - [OpenAPI document](https://skmtc.dev/cekura/apis/cekura-api/revisions/a6fb6a739212?raw)