CEcekura
Evaluators

Run test cases from a JSON spec file

Changed on

Run agent test cases straight from a JSON spec, without creating evaluators. Keep the spec in your repository and post it from CI.

The spec describes only what to test. Pick the target with agent_id and channel on the request — voice for a phone call (default), text for chat, elevenlabs, livekit_v2, or pipecat_v2 for the matching WebRTC connection, or chirp for a raw-PCM WebSocket endpoint — so one file runs unchanged against any agent or environment. LiveKit, Pipecat and CHIRP connection overrides belong in the request's livekit_data, pipecat_data or chirp_data (websocket_url), never in the spec.

Run overrides: the request accepts the same run-level overrides as scenarios_run_voice, applied to every test case in the spec — frequency (multiplies each case's own frequency), agent_number and outbound_phone_number (voice and text; on voice, outbound_phone_number is for inbound agents only), mode (inbound voice agents), concurrency_limit (overrides the spec's defaults.concurrency_limit) and mock_tool_names. Personalities, test profiles and the Cekura-side phone_number are set per test case in the spec.

Pass dry_run=true to validate only: the response describes the run that would happen (test cases, total runs, estimated cost) and nothing is created or charged. Use that on every commit, and the real call on the branches you want tested.

Every problem in a spec is reported at once, keyed by its location in the file (for example scenarios[3].metrics[1]). Metrics must already exist and be enabled for the target agent; personalities and test profiles can be referenced by ID or defined inline.

Fetch the schema this endpoint accepts from scenarios_json_schema.

Cost: voice- or text-simulation rate per run. Test cases from a spec are removed automatically once no run needs them.

post/test_framework/v2/scenarios/run_scenarios_json/

Request

  • Base URL
    https://api.cekura.ai
  • URL
    https://api.cekura.ai/test_framework/v2/scenarios/run_scenarios_json/
  • Auth
    • API key in header X-CEKURA-API-KEY
    • or HTTP bearer
    • or HTTP bearer

Query parameters

dry_runboolean

Validate and price the spec without running or charging anything.

Request body

specunknown
filestring uri

A .json spec file. Use this instead of 'spec' for uploads.

agent_idinteger

Agent to test. Overrides 'agent_id' in the spec, letting one spec file run against several environments.

namestring

Label for this run. Defaults to the spec's suite name.

channel'voice' | 'text' | 'elevenlabs' | 'livekit_v2' | 'pipecat_v2' | 'chirp'

How to reach the agent: 'voice' for a phone call (the default), 'text' for chat, 'elevenlabs' for an ElevenLabs WebRTC session, 'livekit_v2' or 'pipecat_v2' for their respective WebRTC sessions, or 'chirp' for the agent's raw-PCM WebSocket endpoint (CHIRP protocol).

  • voice - voice
  • text - text
  • elevenlabs - elevenlabs
  • livekit_v2 - livekit_v2
  • pipecat_v2 - pipecat_v2
  • chirp - chirp
frequencyinteger

Run every test case this many times, on top of each case's own 'frequency' in the spec. Default 1.

agent_numberstring

Voice and text only. The agent's phone number for this run, in E.164 format, overriding the agent's configured contact number.

outbound_phone_numberstring

Voice and text only. For an inbound voice agent, the number Cekura dials, in E.164 format; defaults to 'agent_number', then the agent's contact number. Rejected for outbound voice agents: set 'phone_number' per test case in the spec instead. For SMS and WhatsApp agents, the contact number for the conversation.

mode'same_number' | 'different_numbers'

Voice, inbound agents only. 'same_number' (default) places every call from the same Cekura number; 'different_numbers' spreads them across the pool and cannot be combined with 'phone_number' in the spec.

  • same_number - same_number
  • different_numbers - different_numbers
concurrency_limitinteger

Cap on the number of runs executed in parallel. Overrides the spec's 'defaults.concurrency_limit'.

calls_per_secondnumber double

Most calls started per second for this run, a number from 0.1 to 4 (e.g. 0.5 starts one call every 2 seconds). Overrides the spec's 'defaults.calls_per_second'. Leave both unset for no limit.

mock_tool_namesstring[] nullable

Names of mock tools to activate for this run. Omit (or send null) to mock every tool configured on the agent; send an empty list to mock none.

livekit_dataobject

Optional LiveKit overrides. These apply only to this run and are not part of the portable test-suite spec.

pipecat_dataobject

Optional Pipecat Cloud overrides. These apply only to this run and are not part of the portable test-suite spec.

chirp_dataobject

Optional CHIRP overrides. 'websocket_url' replaces the agent's stored raw-PCM WebSocket endpoint for this run only — for example a tunnel URL that changes on every CI run. Not part of the portable test-suite spec.

Response

idinteger

ID of the result

agentinteger

ID of the agent

status'running' | 'completed' | 'failed' | 'pending' | 'in_progress' | 'evaluating' | 'in_queue' | 'timeout' | 'cancelled' | 'scaling_up'

Status of the result. Poll until it is completed, failed, timeout or cancelled. It stays non-terminal until every run has ended, then becomes completed if any run completed — even if runs failed their checks or others errored.

run_as_textboolean

Whether the scenario ran as text or not

created_atstring date-time

Example response

{
  "runs": [
    {
      "number": "+11234567890",
      "inbound_number": "+11234567890"
    }
  ],
  "created_at": "2025-02-25T21:00:01.990052Z"
}

Changes