Evaluators

Run evaluators against per-scenario websocket URLs (bulk)

Changed on

WEBSOCKET RUN (per-scenario URLs, bulk): Execute evaluators where each scenario carries its own websocket URL. Useful when you have a fleet of agents or stateful websocket sessions to test in a single batch.

Run-count: total conversations = len(scenarios) × frequency. (Personality and test-profile multipliers are not supported on this endpoint.)

frequency is required on this endpoint (no default).

When to use this vs others (pick the endpoint matching the agent's configured connection — check aiagents_retrieve):

  • run_scenarios_text — single websocket URL or SMS/chat mode (simpler)
  • run_scenarios — real voice / phone / SIP call
  • run_scenarios_pipecat — Pipecat/Daily.co WebRTC deployment

Polling for completion: the response is the result itself — read its top-level id (status: 'running'). Poll GET /test_framework/v1/results/{result_id}/ until status is completed, failed, timeout or cancelled. A result stays non-terminal until every run has ended, then becomes completed if any run completed — even if runs failed their checks or others errored. To gate, require success_runs_count == total_runs_count (failed_runs_count counts only completed runs that failed evaluation).

Known limitation: result.run_type may show 'Text' instead of 'WebSocket' for runs triggered via this endpoint — this is a display issue only and does not affect execution.

post/test_framework/v2/scenarios/run_scenarios_with_websockets/

Request

  • Base URL
    https://api.cekura.ai
  • URL
    https://api.cekura.ai/test_framework/v2/scenarios/run_scenarios_with_websockets/
  • Auth
    • API key in header X-CEKURA-API-KEY
    • or HTTP bearer
    • or HTTP bearer

Request body

agentinteger nullable

Agent to use. Required if none of the scenarios have an agent.

frequencyinteger required

Frequency to run

calls_per_secondnumber double nullable

Most calls started per second for this run, a number from 0.1 to 4 (e.g. 0.5 starts one call every 2 seconds). Leave unset for no limit.

mock_tool_namesstring[] nullable

Names of mock tools to activate for this run. Omit (or send null) to mock every tool configured on the agent; send an empty list to mock none so every tool keeps its original endpoint; send specific names to mock only those. Tools that belong to an MCP server should be selected via the MCP server's name or key, which mocks all of that server's tools as one unit. Legacy MCP sub-tool names are accepted as aliases for the whole parent MCP.

Response

idinteger

ID of the result

agentinteger

ID of the agent

status'running' | 'completed' | 'failed' | 'pending' | 'in_progress' | 'evaluating' | 'in_queue' | 'timeout' | 'cancelled' | 'scaling_up'

Status of the result. Poll until it is completed, failed, timeout or cancelled. It stays non-terminal until every run has ended, then becomes completed if any run completed — even if runs failed their checks or others errored.

run_as_textboolean

Whether the scenario ran as text or not

created_atstring date-time

Example response

{
  "runs": [
    {
      "number": "+11234567890",
      "inbound_number": "+11234567890"
    }
  ],
  "created_at": "2025-02-25T21:00:01.990052Z"
}

Changes