CEcekura
Evaluators

Run evaluators against a VAPI WebRTC agent

Changed on

VAPI WEBRTC VOICE RUN: Execute evaluators against a VAPI agent over WebRTC. The agent (looked up via agent_id or inferred from the first scenario) must have VAPI credentials configured. vapi_assistant_id is optional and defaults to agent.assistant_id.

Cost: voice-simulation rate per run. Run-count math: total runs = len(scenarios) × frequency × max(1, len(personality_ids)) × max(1, len(test_profile_ids)).

When to use this vs others (pick the endpoint matching the agent's configured connection — check aiagents_retrieve):

  • scenarios_run_voice — phone / SIP voice run (no provider WebRTC needed)
  • scenarios_run_text — text/SMS/chat (cheaper)
  • scenarios_run_retell_webrtc — Retell agents over WebRTC
  • scenarios_run_elevenlabs — ElevenLabs Conversational agents
  • scenarios_run_pipecat_v2 — Pipecat Cloud (project credentials, v2)
  • scenarios_run_livekit_v2 — LiveKit (project credentials, v2)
post/test_framework/v2/scenarios/run_scenarios_vapi_webrtc/

Request

  • Base URL
    https://api.cekura.ai
  • URL
    https://api.cekura.ai/test_framework/v2/scenarios/run_scenarios_vapi_webrtc/
  • Auth
    • API key in header X-CEKURA-API-KEY
    • or HTTP bearer
    • or HTTP bearer

Request body

agent_idinteger

ID of the agent to run evaluators for

assistant_idstring

Alternative to agent ID - the assistant ID to use for this scenario Example: "asst_1234567890"

scenariosinteger[]

List of evaluator IDs to run. Either scenarios, tags, or folder_path must be provided. Runs start in the order given. Example: [11, 22, 33]

To run evaluators by tag or folder, use the tags or folder_path fields instead.

tagsstring[]

List of tags to filter evaluators to run. Either evaluators or tags must be provided. Example: ["tag1", "tag2", "tag3"]

folder_pathstring

Dot-separated folder path to select evaluators from (e.g. "Sales.Inbound"). Mutually exclusive with scenarios and tags. Requires project_id.

project_idinteger

Project ID. Required when using folder_path (or when passing a folder path string via scenarios). Example: 1376

frequencyinteger

The number of times each evaluator will run. Each repeat runs the whole list in order before the next one starts (A, B, A, B), rather than grouping repeats of one evaluator. Example: 1

namestring

Label text for result

personality_idsinteger[] nullable

List of personality IDs to override for this run. If not provided, uses the scenario's default personality. Example: [123, 456, 789]

test_profile_idsinteger[] nullable

List of test profile IDs to override for this run. If not provided, uses the scenario's default test profile. Example: [123, 456, 789]

personality_idinteger nullable

Single personality ID override. Merged into personality_ids; prefer personality_ids for multiple.

test_profile_idinteger nullable

Single test profile ID override. Merged into test_profile_ids; prefer test_profile_ids for multiple.

concurrency_limitinteger

Cap on the number of evaluator runs executed in parallel. Default: unbounded (subject to your provider's rate limits). Use this to avoid hitting provider rate limits on large frequency or folder runs. Example: 5

calls_per_secondnumber double nullable

Most calls started per second for this run, a number from 0.1 to 4 (e.g. 0.5 starts one call every 2 seconds). Leave unset for no limit.

mock_tool_namesstring[]

Names of mock tools to activate for this run. Cekura swaps those tools' URLs to mock endpoints while the run is active and restores the originals when it finishes. Omit (or send null) to mock every tool configured on the agent; send an empty list to mock none so every tool keeps its original endpoint. Example: ["get_appointments", "cancel_booking"]

agent_version_idstring

Run against a stored version of the agent instead of the agent as it is live. Cekura makes a private copy of the agent in your provider account, puts that version's prompt on it, and runs against the copy — voice, model and tools stay as the agent is configured now, and the live agent is never changed. Accepts the version label shown in the agent's version history (v12, or p1 for a version an improvement run proposed) or its numeric history_id. List them with GET /test_framework/v1/aiagents/{id}/history/; a version is usable when its is_runnable is true. Available for VAPI, Retell, ElevenLabs and Bland agents that are outbound and have a single prompt. Example: "v12"

vapi_assistant_idstring

VAPI assistant ID to test against. Overrides the agent's stored assistant_id for this run.

Response

idinteger

ID of the result

agentinteger

ID of the agent

status'running' | 'completed' | 'failed' | 'pending' | 'in_progress' | 'evaluating' | 'in_queue' | 'timeout' | 'cancelled' | 'scaling_up'

Status of the result. Poll until it is completed, failed, timeout or cancelled. It stays non-terminal until every run has ended, then becomes completed if any run completed — even if runs failed their checks or others errored.

run_as_textboolean

Whether the scenario ran as text or not

created_atstring date-time

Example response

{
  "runs": [
    {
      "number": "+11234567890",
      "inbound_number": "+11234567890"
    }
  ],
  "created_at": "2025-02-25T21:00:01.990052Z"
}

Changes