CEcekura
Scenarios

Create an evaluator

Changed on

scenarios_create: Create a test scenario (evaluator) that simulates a caller interacting with your agent. Required: name, instructions or program (see modes), and an agent or project to attach to.

Prefer generation for instruction scenarios: scenarios_generate_bg grounds them in the agent's description, knowledge base and mock tools, and auto-attaches a test profile and the project's metrics — even for a single scenario (num_scenarios: 1, with the case described in extra_instructions). Use this create tool for conditional_actions scenarios whose structure is exactly known, or to save instruction text the user supplied verbatim — not to hand-write behavioral instructions a generation call could produce.

Two modes — pick one via scenario_type:

  • instruction — plain-text caller instructions in the instructions field. The simulated caller uses an LLM to paraphrase your instructions during the call. Prefer scenarios_generate_bg for these (see above).

  • conditional_actions — deterministic, rule-based caller. Provide the structured conditional_actions object: an optional first_message, then conditions of when / say / then. Steps are spoken verbatim unless wrapped in <ai_generated>...</ai_generated>. Use either conditional_actions or instructions — not both.

Common companion fields:

  • personality (required) — caller voice/personality ID. Browse via GET /personalities/. It also sets the caller's spoken language.
  • expected_outcome_prompt — what a passing call looks like, graded statement by statement after each run. It is evaluated ONLY when the project's Expected Outcome metric is attached to the scenario, so attach that metric every time you set this field.
  • metrics — list of metric IDs to evaluate after each run. A scenario created here starts with NO metrics attached (unlike generated ones): always attach the project's baseline set, including the Expected Outcome metric.
  • tags — for grouping and later selection via run_scenarios?tags=[...].
  • folder_path — e.g. "Sales.Inbound" for folder organization.
  • test_profile — a test profile ID providing caller metadata (order_id, user_email, etc.).

To run evaluators after creating: call run_scenarios (voice), run_scenarios_text, run_scenarios_with_websockets, or run_scenarios_pipecat with the scenario ID(s) returned here.

post/test_framework/v2/scenarios/

Request

  • Base URL
    https://api.cekura.ai
  • URL
    https://api.cekura.ai/test_framework/v2/scenarios/
  • Auth
    • API key in header X-CEKURA-API-KEY
    • or HTTP bearer
    • or HTTP bearer

Request body

agentinteger nullable

ID of the agent. Either agent or project must be provided.

projectinteger nullable

ID of the project. Either agent or project must be provided.

assistant_idstring

Alternative to agent ID - the assistant ID to use for this scenario Example: "asst_1234567890"

namestring required

Name of the evaluator

personalityinteger required

The personality of the evaluator

scenario_type'instruction' | 'conditional_actions'

Type of scenario to create:

  • instruction: Standard instruction-based scenario where the instructions field contains plain text
  • conditional_actions: Conditional action-based scenario where the instructions field contains a JSON structure

Default: instruction. Must be sent as conditional_actions when the request includes the conditional_actions field.

  • instruction - Instruction
  • conditional_actions - Conditional Actions
instructionsstring

Scenario Instructions - the format depends on the scenario_type:

For scenario_type="instruction": Provide plain text instructions that describe the scenario behavior. Example: "You are a customer calling to inquire about your order status. Be polite and provide order number 12345 when asked."

For scenario_type="conditional_actions": Use the structured conditional_actions field instead. If you send via instructions, pass a JSON-formatted string with the following structure:

{
  "role": "A friendly customer calling about an appointment",
  "first_message": "Hi, I'd like to check on my upcoming appointment <silence time=\"1.5s\" />",
  "conditions": [
    {"when": "The agent asks for your name", "say": "My name is Sarah Johnson"},
    {"when": "The agent asks for your date of birth", "say": "January first, nineteen ninety"},
    {
      "when": "The agent confirms your identity and provides appointment details",
      "say": "Thank you, that's all I needed <silence time=\"1.0s\" />",
      "then": ["<endcall />"]
    }
  ]
}

Note: When sent via instructions, the value must be a JSON-formatted string (e.g. JSON.stringify(...) / json.dumps(...)). For new integrations, use the structured conditional_actions field below. Responses return instructions in this same shape.

Structure Details:

  • role (string): Personality/role description for the scenario actor

  • first_message (string | object): What the scenario actor says first, always verbatim. Empty means wait for the agent to speak. Use {"say": "...", "then": [...]} to add follow-up steps after it.

  • conditions (array): Ordered list of conditions that define the conversation flow

    Each condition object has:

    • when (string): Natural-language description of the main agent's observable action, written from a third-person observer perspective (e.g. "The agent asks for your date of birth", "The agent confirms the cancellation"). Matched semantically against the conversation context.

    • say (string): What the scenario actor says or does, spoken verbatim. Wrap the whole step in <ai_generated>...</ai_generated> to have the actor improvise from it instead.

      • SSML pause tags: <silence time="1.5s" /> (interruptible) or <hold time="2s" /> (not interruptible). <silence> lets the main agent jump in mid-pause; <hold> enforces dead air.
      • Emotion markers: [laughter], [sigh], etc. (can repeat like [laughter] [laughter])
      • Special actions: <endcall /> to end the call
      • Action tags only work in verbatim steps, not inside <ai_generated>.

    • then (array of strings, optional): Follow-up steps in order. Each fires on the actor's next turn after one main-agent reply, regardless of its content. <interruption> is only allowed here.

Usage Tips:

  • Write when as third-person descriptions of what the main agent does (e.g. "The agent confirms the cancellation")
  • End with an <endcall /> step for clean call termination
  • Combine SSML tags and emotion markers naturally: "Great <silence time=\"1.5s\" /> [laughter] [laughter]"
expected_outcome_promptstring

Expected Outcome Prompt Example: "The user should be able to complete the order"

metricsinteger[]

List of metric IDs to associate with this evaluator Example: [123, 456, 789]

tagsstring[]

List of tags to associate with the evaluator Example: ["tag1", "tag2", "tag3"]

tool_idsstring[]

List of tool IDs to use for evaluator

test_profileinteger

The test profile ID to use for the evaluator

phone_numberinteger

The phone number ID to use for the evaluator (works for both inbound and outbound)

inbound_phone_numberinteger

(Deprecated) The inbound phone number ID to use for the evaluator. Use phone_number instead, which works for both inbound and outbound.

folder_pathstring nullable

Dot-separated path of the folder to assign this evaluator to (e.g. "Sales.Inbound"). Folders must be created before use — they are not auto-created. Use scenarios_create_folder_create to create each level before assigning evaluators. Example: to use "Sales.Inbound", first create "Sales", then create "Inbound" inside it.

generated_mock_tool_entriesunknown
first_messagestring

Opening line the simulated caller says to the agent. Defaults to "Hello".

max_durationinteger nullable

Max call duration in seconds for this scenario. If not provided or set to null, the scenario will use the project's max_call_duration setting. Valid range: 10-3600 seconds (10 seconds to 60 minutes) Example: 600 (10 minutes)

scenario_language'af' | 'ar' | 'bn' | 'bg' | 'zh' | 'cs' | 'da' | 'nl' | 'en' | 'et' | 'fi' | 'fr' | 'de' | 'el' | 'gu' | 'hi' | 'he' | 'hu' | 'id' | 'it' | 'ja' | 'kn' | 'kk' | 'ko' | 'ms' | 'ml' | 'mr' | 'multi' | 'no' | 'pl' | 'pa' | 'pt' | 'ro' | 'ru' | 'sk' | 'es' | 'sv' | 'th' | 'tr' | 'tl' | 'ta' | 'te' | 'uk' | 'uz' | 'vi'

Language of the evaluator. Must match the chosen personality's language — mismatches are rejected.

  • af - Afrikaans
  • ar - Arabic
  • bn - Bengali
  • bg - Bulgarian
  • zh - Chinese Simplified
  • cs - Czech
  • da - Danish
  • nl - Dutch
  • en - English
  • et - Estonian
  • fi - Finnish
  • fr - French
  • de - German
  • el - Greek
  • gu - Gujarati
  • hi - Hindi
  • he - Hebrew
  • hu - Hungarian
  • id - Indonesian
  • it - Italian
  • ja - Japanese
  • kn - Kannada
  • kk - Kazakh
  • ko - Korean
  • ms - Malay
  • ml - Malayalam
  • mr - Marathi
  • multi - Multilingual
  • no - Norwegian
  • pl - Polish
  • pa - Punjabi
  • pt - Portuguese
  • ro - Romanian
  • ru - Russian
  • sk - Slovak
  • es - Spanish
  • sv - Swedish
  • th - Thai
  • tr - Turkish
  • tl - Tagalog
  • ta - Tamil
  • te - Telugu
  • uk - Ukrainian
  • uz - Uzbek
  • vi - Vietnamese

Example request

{
  "tool_ids": [
    "TOOL_DTMF",
    "TOOL_END_CALL",
    "TOOL_END_CALL_ONLY_ON_TRANSFER"
  ]
}

Response

idinteger
namestring

Name of the evaluator

agentinteger

Agent ID Example: 2142

projectinteger nullable
assistant_idstring

Alternative to agent ID - the assistant ID to use for this scenario Example: "asst_1234567890"

personalityinteger

ID of the personality

personality_namestring

Name of the personality

agent_namestring

Name of the agent associated with this scenario Example: "Customer Support Agent"

tagsstring[]

List of tags of the evaluators Example: ["tag1", "tag2", "tag3"]

tool_idsunknown
metricsinteger[]

List of metric IDs Example: [123, 456, 789]

metric_namesstring[]

List of metric names Example: ["Metric 1", "Metric 2", "Metric 3"]

phone_numberstring

Phone number eg: +17776666333

first_messagestring

First message of the evaluator

inbound_phone_numberinteger nullable
instructionsstring

Scenario Instructions - format depends on scenario_type:

  • For instruction type: Plain text instructions
  • For conditional_actions type: JSON string with role, first_message and a conditions array. Each condition has when (a natural-language third-person description of the main agent's observable action — e.g. "The agent asks for your date of birth", matched semantically), say (spoken verbatim, with optional <silence time="1.5s" /> (interruptible) or <hold time="2s" /> (not interruptible) and [laughter] tags; wrap the whole step in <ai_generated>...</ai_generated> to improvise instead) and optional then follow-up steps. Responses return this format unless conditional_actions_format=legacy is passed. The earlier id-based format (id, condition, action, type, fixed_message) is still accepted on writes.
conditional_actionsunknown
condition_audioobject

Pre-recorded clips available to this scenario's conditional actions, keyed by clip id. Read-only: upload, replace and delete clips, and correct a transcript, through condition-audio/. Reference a clip from any number of verbatim steps by pasting its tag; steps lists where it is used (e.g. conditions[1].then[0]).

Example:

{
  "greeting": {"tag": "<audio id=\"greeting\"/>", "filename": "greeting.wav",
               "status": "ready", "duration": 2.4, "transcript": "Hi, is this the clinic?",
               "steps": ["first_message", "conditions[1]"]}
}
simulation_descriptionstring
information_fieldsobject

Information fields of the evaluator Example:

{
    "user_name": "John Doe",
    "user_email": "john.doe@example.com",
}
expected_outcome_promptstring

Expected outcome prompt of the evaluator Example: "The user should be able to complete the order"

scenario_language'af' | 'ar' | 'bn' | 'bg' | 'zh' | 'cs' | 'da' | 'nl' | 'en' | 'et' | 'fi' | 'fr' | 'de' | 'el' | 'gu' | 'hi' | 'he' | 'hu' | 'id' | 'it' | 'ja' | 'kn' | 'kk' | 'ko' | 'ms' | 'ml' | 'mr' | 'multi' | 'no' | 'pl' | 'pa' | 'pt' | 'ro' | 'ru' | 'sk' | 'es' | 'sv' | 'th' | 'tr' | 'tl' | 'ta' | 'te' | 'uk' | 'uz' | 'vi'

Language of the scenario

  • af - Afrikaans
  • ar - Arabic
  • bn - Bengali
  • bg - Bulgarian
  • zh - Chinese Simplified
  • cs - Czech
  • da - Danish
  • nl - Dutch
  • en - English
  • et - Estonian
  • fi - Finnish
  • fr - French
  • de - German
  • el - Greek
  • gu - Gujarati
  • hi - Hindi
  • he - Hebrew
  • hu - Hungarian
  • id - Indonesian
  • it - Italian
  • ja - Japanese
  • kn - Kannada
  • kk - Kazakh
  • ko - Korean
  • ms - Malay
  • ml - Malayalam
  • mr - Marathi
  • multi - Multilingual
  • no - Norwegian
  • pl - Polish
  • pa - Punjabi
  • pt - Portuguese
  • ro - Romanian
  • ru - Russian
  • sk - Slovak
  • es - Spanish
  • sv - Swedish
  • th - Thai
  • tr - Turkish
  • tl - Tagalog
  • ta - Tamil
  • te - Telugu
  • uk - Ukrainian
  • uz - Uzbek
  • vi - Vietnamese
scenario_typestring

Type of scenario:

  • instruction: Standard instruction-based scenario (plain text instructions)
  • conditional_actions: Conditional action-based scenario (JSON with role, conditions, SSML tags, emotion markers)
suite_type'undefined' | 'infrastructure'

Type of predefined suite (e.g., infrastructure)

  • undefined - Undefined
  • infrastructure - Infrastructure
is_predefinedboolean

Whether this is a predefined scenario (template)

test_profileinteger nullable
dynamic_variable_valuesunknown
generated_mock_tool_entriesunknown
folder_pathstring nullable

Path of an existing folder to assign this evaluator to, e.g. "Sales" or "Sales.Inbound". Create it first with the create-folder endpoint.

max_durationinteger nullable

Max call duration in seconds for this scenario. If not provided or set to null, the scenario will use the project's max_call_duration setting. Valid range: 10-3600 seconds (10 seconds to 60 minutes) Example: 600 (10 minutes)

Changes