--- title: "Run evaluators against an agent over any transport" method: POST path: "/test_framework/v2/scenarios/run/" tags: ["Evaluators"] --- # Run evaluators against an agent over any transport `POST /test_framework/v2/scenarios/run/` ## Request - Base URL: `https://api.cekura.ai` - URL: `https://api.cekura.ai/test_framework/v2/scenarios/run/` - Auth: one of: - API key in header `X-CEKURA-API-KEY` - HTTP bearer - HTTP bearer Start evaluator runs against an agent. Pick how the simulated user reaches the agent with `transport`; it must be one of the agent's `enabled_connections` (see `aiagents_retrieve`), and a transport that is not enabled is rejected with the list of enabled ones. Fields that belong to another transport are rejected. **Scenario selection (pick one):** `scenarios` (evaluator IDs), `tags`, or `folder_path` with `project_id`. `websocket` and `pipecat_v1` take their evaluators inside `websocket_urls` / `pipecat_rooms` instead, one entry per evaluator with its own URL. **Run-count math:** total runs = `len(scenarios) × frequency × max(1, len(personality_ids)) × max(1, len(test_profile_ids))`. `text`, `email` and `websocket` runs bill at the text-simulation rate; every other transport places a voice call and bills at the voice rate. **Polling for completion:** the response is the result itself — read its top-level `id`. Poll `GET /test_framework/v1/results/{result_id}/` until `status` is `completed`, `failed`, `timeout` or `cancelled`. Check `failed_reasons` for infrastructure errors and `runs[].error_message` for per-run errors. ## Request body - RunScenariosRequest - `transport` 'voice' | 'sip' | 'text' | 'email' | 'vapi_webrtc' | 'retell_webrtc' | 'elevenlabs' | 'livekit_v2' | 'pipecat_v2' | 'chirp' | 'agora' | 'websocket' | 'pipecat_v1', required: How the simulated user reaches the agent. Must be enabled on the agent; `aiagents_retrieve` lists the options under `enabled_connections`. `websocket` and `pipecat_v1` carry their connection details per scenario instead. * `voice` - Voice * `sip` - SIP * `text` - Text * `email` - Email * `vapi_webrtc` - VAPI WebRTC * `retell_webrtc` - Retell WebRTC * `elevenlabs` - ElevenLabs * `livekit_v2` - LiveKit * `pipecat_v2` - Pipecat Cloud * `chirp` - Raw-PCM WebSocket * `agora` - Agora * `websocket` - WebSocket (per-scenario URLs) * `pipecat_v1` - Pipecat (per-scenario room URLs) - `agent` integer: Agent ID to run against. Required unless every selected evaluator already belongs to an agent. - `scenarios` integer[]: Evaluator IDs to run, in order. Provide exactly one of `scenarios`, `tags`, `folder_path`. The dashboard may also send its selection shorthand (`"all"`, a count, or a folder path). - `tags` string[]: Run every evaluator carrying any of these tags. - `folder_path` string: Dot-separated folder path whose evaluators run. Requires `project_id`. - `project_id` integer: Project ID. Required with `folder_path`. - `frequency` integer: Times each evaluator runs. Total runs = evaluators × frequency × personalities × test profiles. - `name` string: Label for the result. - `personality_ids` integer[], nullable: Personality IDs to run each evaluator with instead of its own. Only personalities enabled for the project. - `test_profile_ids` integer[], nullable: Test profile IDs to run each evaluator with instead of its own. - `concurrency_limit` integer, nullable: Cap on runs executing in parallel. Default: unbounded. - `calls_per_second` number, double, nullable: Most calls started per second for this run, a number from 0.1 to 4 (e.g. 0.5 starts one call every 2 seconds). Leave unset for no limit. - `mock_tool_names` string[], nullable: Mock tools to activate for this run. Omit to mock every configured tool; send an empty list to mock none. - `agent_version_id` string: Version label or numeric history id of a stored agent version to run against instead of the live agent. - `outbound_phone_number` string: `voice`, `text`: override the number dialed, in E.164 format. - `agent_number` string: `voice`, `text`: override the number the agent under test receives calls on, in E.164 format. - `mode` 'same_number' | 'different_numbers': `voice`: use one phone number for every run (`same_number`) or rotate numbers (`different_numbers`). * `same_number` - same_number * `different_numbers` - different_numbers - `sip_endpoint` string: `sip`: override the agent's SIP endpoint for this run. - `websocket_url` string: `text`: websocket URL the simulated user connects to instead of the agent's chat gateway. `chirp`: override the agent's raw-PCM websocket endpoint. - `websocket_headers` object: `text`: headers sent on the websocket connection, merged over the agent's configured headers. - `agent_email` string, email: `email`: override the agent's configured email address for this run. - `vapi_assistant_id` string: `vapi_webrtc`: VAPI assistant ID to test. Defaults to the agent's stored assistant id. - `retell_agent_id` string: `retell_webrtc`: Retell agent ID to test. Defaults to the agent's stored assistant id. - `elevenlabs_agent_id` string: `elevenlabs`: ElevenLabs agent ID to test. Defaults to the agent's stored assistant id. - `elevenlabs_data` SchemaElevenLabsData: Per-run ElevenLabs overrides. Credentials are read from the agent. - `elevenlabs_base_url_override` string, nullable: Override the ElevenLabs base URL for this run (e.g. a regional or self-hosted endpoint). - `elevenlabs_environment` string, nullable: ElevenLabs environment for this run, used to resolve the agent's environment variables (tool URLs, secrets, headers). Defaults to the environment saved on the agent, then to production. - `elevenlabs_branch_id` string, nullable: ElevenLabs agent branch for this run. Defaults to the branch saved on the agent, then to the agent's default branch. - `livekit_data` LivekitRunDataOverride: Per-run LiveKit overrides. Credentials (api_key/api_secret) are never taken here — they are read from the agent/project ProviderCredential. - `config` unknown - `agent_name` string, nullable: LiveKit agent name for explicit dispatch. Overrides the agent/project saved agent_name. - `url` string, nullable: LiveKit server URL override. Falls back to the agent/project saved url when omitted. - `pipecat_data` SchemaPipecatV2Data: Per-run Pipecat Cloud overrides. Credentials are read from the agent/project ProviderCredential. - `pipecat_agent_name` string, nullable: Pipecat Cloud agent name. Overrides the agent/project saved pipecat_agent_name. - `config` unknown - `room_properties` unknown - `websocket_urls` WebsocketScenarioEntry[]: `websocket`: the evaluators to run, each with its own websocket URL. Replaces `scenarios`. - `scenario` integer, required: Evaluator ID to run. - `websocket_url` string, required: ws:// or wss:// URL this evaluator connects to. - `pipecat_rooms` PipecatRoomEntry[]: `pipecat_v1`: the evaluators to run, each with its own Daily.co room URL. Replaces `scenarios`. - `scenario` integer, required: Evaluator ID to run. - `pipecat_room_url` string, required: Daily.co room URL this evaluator joins. - `pipecat_token` string: Room token, when the room requires one. - `call_id` string: Call ID to associate with the Pipecat session. - `publish_data_message` unknown ## Response `200` - RunScenarioResult - `id` integer: Unique identifier for the Result Example: `123` - `name` string, required: Name of the Result Example: `"Test Result 1"` - `agent` integer: ID of the AI agent that was tested Example: `123` - `agent_inbound` boolean: Whether the agent is configured for inbound calls (true) or outbound calls (false) Example: `true` or `false` - `status` 'running' | 'completed' | 'failed' | 'pending' | 'in_progress' | 'evaluating' | 'in_queue' | 'timeout' | 'cancelled' | 'scaling_up': Current status of the result * `running` - Running * `completed` - Completed * `failed` - Failed * `pending` - Pending * `in_progress` - In Progress * `evaluating` - Evaluating * `in_queue` - In Queue * `timeout` - Timeout * `cancelled` - Cancelled * `scaling_up` - Scaling Up - `success_rate` number, double: Percentage of runs that passed, 0 to 100 - `run_as_text` boolean: Whether this test was run in text mode instead of voice mode Example: `true` or `false` - `is_cronjob` string: Whether this result was created by a scheduled cronjob Example: `true` or `false` - `runs` object: Run summaries for this result. For full run data including transcripts and metric scores, fetch the result detail endpoint — there `runs` is a dict keyed by run ID. - `created_at` string, date-time: Timestamp when this test result was created Example: `2021-01-01 00:00:00` - `updated_at` string, date-time: Timestamp when this test result was last updated ## Other responses - `400` ## Changes - **2026-11-03** `a6fb6a739212`: 1 info - endpoint added [Operation history](https://skmtc.dev/cekura/apis/cekura-api/changes/test_framework/v2/scenarios/run/post.md) --- [API](https://skmtc.dev/cekura/apis/cekura-api.md) - [All operations](https://skmtc.dev/cekura/apis/cekura-api/llms.txt) - [OpenAPI document](https://skmtc.dev/cekura/apis/cekura-api/revisions/a6fb6a739212?raw)