Evaluators

Run evaluators against an email agent

Changed on

EMAIL RUN: Execute evaluators against an agent that communicates over email. Cekura sends an email to the agent's address and simulates the full conversation, receiving replies via inbound webhook.

The agent must have an email_id configured, or you can supply agent_email as a one-time override. The agent_email field takes precedence if both are set.

Run-count: total conversations = len(scenarios) × frequency.

Inbound agents (agent.inbound=true): Cekura sends the first email to the agent. Outbound agents (agent.inbound=false): the response includes inbound_email — the address your agent should email first to start the conversation.

Polling for completion: the response is the result itself — read its top-level id (status: 'running'). Poll GET /test_framework/v1/results/{result_id}/ until status is completed, failed, timeout or cancelled. A result stays non-terminal until every run has ended, then becomes completed if any run completed — even if runs failed their checks or others errored. To gate, require success_runs_count == total_runs_count (failed_runs_count counts only completed runs that failed evaluation).

post/test_framework/v1/scenarios/run_scenarios_email/

Request

  • Base URL
    https://api.cekura.ai
  • URL
    https://api.cekura.ai/test_framework/v1/scenarios/run_scenarios_email/
  • Auth
    • API key in header X-CEKURA-API-KEY
    • or HTTP bearer
    • or HTTP bearer

Request body

agent_idinteger nullable

ID of the agent to run against. Required if none of the scenarios have an agent.

scenariosinteger[] required

List of evaluator IDs to run.

frequencyinteger required

Number of times to run each scenario.

calls_per_secondnumber double nullable

Most calls started per second for this run, a number from 0.1 to 4 (e.g. 0.5 starts one call every 2 seconds). Leave unset for no limit.

agent_emailstring email nullable

Override the agent's configured email address for this run.

Response

idinteger

Unique identifier for the Result Example: 123

namestring required

Name of the Result Example: "Test Result 1"

agentinteger

ID of the AI agent that was tested Example: 123

agent_inboundboolean

Whether the agent is configured for inbound calls (true) or outbound calls (false) Example: true or false

status'running' | 'completed' | 'failed' | 'pending' | 'in_progress' | 'evaluating' | 'in_queue' | 'timeout' | 'cancelled' | 'scaling_up'

Current status of the result

  • running - Running
  • completed - Completed
  • failed - Failed
  • pending - Pending
  • in_progress - In Progress
  • evaluating - Evaluating
  • in_queue - In Queue
  • timeout - Timeout
  • cancelled - Cancelled
  • scaling_up - Scaling Up
success_ratenumber double

Percentage of runs that passed, 0 to 100

run_as_textboolean

Whether this test was run in text mode instead of voice mode Example: true or false

is_cronjobstring

Whether this result was created by a scheduled cronjob Example: true or false

runsobject

Run summaries for this result. For full run data including transcripts and metric scores, fetch the result detail endpoint — there runs is a dict keyed by run ID.

created_atstring date-time

Timestamp when this test result was created Example: 2021-01-01 00:00:00

updated_atstring date-time

Timestamp when this test result was last updated

Changes