CEcekura
Evaluators

Run evaluators against an agent over any transport

Changed on

Start evaluator runs against an agent. Pick how the simulated user reaches the agent with transport; it must be one of the agent's enabled_connections (see aiagents_retrieve), and a transport that is not enabled is rejected with the list of enabled ones. Fields that belong to another transport are rejected.

Scenario selection (pick one): scenarios (evaluator IDs), tags, or folder_path with project_id. websocket and pipecat_v1 take their evaluators inside websocket_urls / pipecat_rooms instead, one entry per evaluator with its own URL.

Run-count math: total runs = len(scenarios) × frequency × max(1, len(personality_ids)) × max(1, len(test_profile_ids)). text, email and websocket runs bill at the text-simulation rate; every other transport places a voice call and bills at the voice rate.

Polling for completion: the response is the result itself — read its top-level id. Poll GET /test_framework/v1/results/{result_id}/ until status is completed, failed, timeout or cancelled. Check failed_reasons for infrastructure errors and runs[].error_message for per-run errors.

post/test_framework/v2/scenarios/run/

Request

  • Base URL
    https://api.cekura.ai
  • URL
    https://api.cekura.ai/test_framework/v2/scenarios/run/
  • Auth
    • API key in header X-CEKURA-API-KEY
    • or HTTP bearer
    • or HTTP bearer

Request body

transport'voice' | 'sip' | 'text' | 'email' | 'vapi_webrtc' | 'retell_webrtc' | 'elevenlabs' | 'livekit_v2' | 'pipecat_v2' | 'chirp' | 'agora' | 'websocket' | 'pipecat_v1' required

How the simulated user reaches the agent. Must be enabled on the agent; aiagents_retrieve lists the options under enabled_connections. websocket and pipecat_v1 carry their connection details per scenario instead.

  • voice - Voice
  • sip - SIP
  • text - Text
  • email - Email
  • vapi_webrtc - VAPI WebRTC
  • retell_webrtc - Retell WebRTC
  • elevenlabs - ElevenLabs
  • livekit_v2 - LiveKit
  • pipecat_v2 - Pipecat Cloud
  • chirp - Raw-PCM WebSocket
  • agora - Agora
  • websocket - WebSocket (per-scenario URLs)
  • pipecat_v1 - Pipecat (per-scenario room URLs)
agentinteger

Agent ID to run against. Required unless every selected evaluator already belongs to an agent.

scenariosinteger[]

Evaluator IDs to run, in order. Provide exactly one of scenarios, tags, folder_path. The dashboard may also send its selection shorthand ("all", a count, or a folder path).

tagsstring[]

Run every evaluator carrying any of these tags.

folder_pathstring

Dot-separated folder path whose evaluators run. Requires project_id.

project_idinteger

Project ID. Required with folder_path.

frequencyinteger

Times each evaluator runs. Total runs = evaluators × frequency × personalities × test profiles.

namestring

Label for the result.

personality_idsinteger[] nullable

Personality IDs to run each evaluator with instead of its own. Only personalities enabled for the project.

test_profile_idsinteger[] nullable

Test profile IDs to run each evaluator with instead of its own.

concurrency_limitinteger nullable

Cap on runs executing in parallel. Default: unbounded.

calls_per_secondnumber double nullable

Most calls started per second for this run, a number from 0.1 to 4 (e.g. 0.5 starts one call every 2 seconds). Leave unset for no limit.

mock_tool_namesstring[] nullable

Mock tools to activate for this run. Omit to mock every configured tool; send an empty list to mock none.

agent_version_idstring

Version label or numeric history id of a stored agent version to run against instead of the live agent.

outbound_phone_numberstring

voice, text: override the number dialed, in E.164 format.

agent_numberstring

voice, text: override the number the agent under test receives calls on, in E.164 format.

mode'same_number' | 'different_numbers'

voice: use one phone number for every run (same_number) or rotate numbers (different_numbers).

  • same_number - same_number
  • different_numbers - different_numbers
sip_endpointstring

sip: override the agent's SIP endpoint for this run.

websocket_urlstring

text: websocket URL the simulated user connects to instead of the agent's chat gateway. chirp: override the agent's raw-PCM websocket endpoint.

websocket_headersobject

text: headers sent on the websocket connection, merged over the agent's configured headers.

agent_emailstring email

email: override the agent's configured email address for this run.

vapi_assistant_idstring

vapi_webrtc: VAPI assistant ID to test. Defaults to the agent's stored assistant id.

retell_agent_idstring

retell_webrtc: Retell agent ID to test. Defaults to the agent's stored assistant id.

elevenlabs_agent_idstring

elevenlabs: ElevenLabs agent ID to test. Defaults to the agent's stored assistant id.

Response

idinteger

Unique identifier for the Result Example: 123

namestring required

Name of the Result Example: "Test Result 1"

agentinteger

ID of the AI agent that was tested Example: 123

agent_inboundboolean

Whether the agent is configured for inbound calls (true) or outbound calls (false) Example: true or false

status'running' | 'completed' | 'failed' | 'pending' | 'in_progress' | 'evaluating' | 'in_queue' | 'timeout' | 'cancelled' | 'scaling_up'

Current status of the result

  • running - Running
  • completed - Completed
  • failed - Failed
  • pending - Pending
  • in_progress - In Progress
  • evaluating - Evaluating
  • in_queue - In Queue
  • timeout - Timeout
  • cancelled - Cancelled
  • scaling_up - Scaling Up
success_ratenumber double

Percentage of runs that passed, 0 to 100

run_as_textboolean

Whether this test was run in text mode instead of voice mode Example: true or false

is_cronjobstring

Whether this result was created by a scheduled cronjob Example: true or false

runsobject

Run summaries for this result. For full run data including transcripts and metric scores, fetch the result detail endpoint — there runs is a dict keyed by run ID.

created_atstring date-time

Timestamp when this test result was created Example: 2021-01-01 00:00:00

updated_atstring date-time

Timestamp when this test result was last updated

Changes