---
title: "Create an evaluator"
method: POST
path: "/test_framework/v2/scenarios/"
tags: ["Scenarios"]
---

# Create an evaluator

`POST /test_framework/v2/scenarios/`

## Request

- Base URL: `https://api.cekura.ai`
- URL: `https://api.cekura.ai/test_framework/v2/scenarios/`
- Auth: one of:
  - API key in header `X-CEKURA-API-KEY`
  - HTTP bearer
  - HTTP bearer

scenarios_create: Create a test scenario (evaluator) that simulates a caller interacting with your agent. Required: `name`, `instructions` or program (see modes), and an `agent` or `project` to attach to.

**Prefer generation for instruction scenarios:** `scenarios_generate_bg` grounds them in the agent's description, knowledge base and mock tools, and auto-attaches a test profile and the project's metrics — even for a single scenario (`num_scenarios: 1`, with the case described in `extra_instructions`). Use this create tool for `conditional_actions` scenarios whose structure is exactly known, or to save instruction text the user supplied verbatim — not to hand-write behavioral instructions a generation call could produce.

**Two modes — pick one via `scenario_type`:**

- `instruction` — plain-text caller instructions in the `instructions` field. The simulated caller uses an LLM to paraphrase your instructions during the call. Prefer `scenarios_generate_bg` for these (see above).

- `conditional_actions` — deterministic, rule-based caller. Provide the structured `conditional_actions` object: an optional `first_message`, then `conditions` of `when` / `say` / `then`. Steps are spoken verbatim unless wrapped in `<ai_generated>...</ai_generated>`. Use either `conditional_actions` or `instructions` — not both.

**Common companion fields:**
- `personality` (required) — caller voice/personality ID. Browse via `GET /personalities/`. It also sets the caller's spoken language.
- `expected_outcome_prompt` — what a passing call looks like, graded statement by statement after each run. It is evaluated ONLY when the project's Expected Outcome metric is attached to the scenario, so attach that metric every time you set this field.
- `metrics` — list of metric IDs to evaluate after each run. A scenario created here starts with NO metrics attached (unlike generated ones): always attach the project's baseline set, including the Expected Outcome metric.
- `tags` — for grouping and later selection via `run_scenarios?tags=[...]`.
- `folder_path` — e.g. `"Sales.Inbound"` for folder organization.
- `test_profile` — a test profile ID providing caller metadata (order_id, user_email, etc.).

**To run evaluators after creating:** call `run_scenarios` (voice), `run_scenarios_text`, `run_scenarios_with_websockets`, or `run_scenarios_pipecat` with the scenario ID(s) returned here.

## Request body

- SchemaPostScenario
  - `agent` integer, nullable: ID of the agent. Either agent or project must be provided.
  - `project` integer, nullable: ID of the project. Either agent or project must be provided.
  - `assistant_id` string: Alternative to agent ID - the assistant ID to use for this scenario Example: `"asst_1234567890"`
  - `name` string, required: Name of the evaluator
  - `personality` integer, required: The personality of the evaluator
  - `scenario_type` 'instruction' | 'conditional_actions': Type of scenario to create: - `instruction`: Standard instruction-based scenario where the instructions field contains plain text - `conditional_actions`: Conditional action-based scenario where the instructions field contains a JSON structure Default: `instruction`. Must be sent as `conditional_actions` when the request includes the `conditional_actions` field. * `instruction` - Instruction * `conditional_actions` - Conditional Actions
  - `instructions` string: Scenario Instructions - the format depends on the scenario_type: **For scenario_type="instruction":** Provide plain text instructions that describe the scenario behavior. Example: `"You are a customer calling to inquire about your order status. Be polite and provide order number 12345 when asked."` **For scenario_type="conditional_actions":** Use the structured `conditional_actions` field instead. If you send via `instructions`, pass a JSON-formatted string with the following structure: ```json { "role": "A friendly customer calling about an appointment", "first_message": "Hi, I'd like to check on my upcoming appointment <silence time=\"1.5s\" />", "conditions": [ {"when": "The agent asks for your name", "say": "My name is Sarah Johnson"}, {"when": "The agent asks for your date of birth", "say": "January first, nineteen ninety"}, { "when": "The agent confirms your identity and provides appointment details", "say": "Thank you, that's all I needed <silence time=\"1.0s\" />", "then": ["<endcall />"] } ] } ``` **Note:** When sent via `instructions`, the value must be a JSON-formatted string (e.g. `JSON.stringify(...)` / `json.dumps(...)`). For new integrations, use the structured `conditional_actions` field below. Responses return `instructions` in this same shape. **Structure Details:** - **`role`** (string): Personality/role description for the scenario actor - **`first_message`** (string | object): What the scenario actor says first, always verbatim. Empty means wait for the agent to speak. Use `{"say": "...", "then": [...]}` to add follow-up steps after it. - **`conditions`** (array): Ordered list of conditions that define the conversation flow **Each condition object has:** - **`when`** (string): Natural-language description of the main agent's observable action, written from a third-person observer perspective (e.g. `"The agent asks for your date of birth"`, `"The agent confirms the cancellation"`). Matched semantically against the conversation context. - **`say`** (string): What the scenario actor says or does, spoken verbatim. Wrap the whole step in `<ai_generated>...</ai_generated>` to have the actor improvise from it instead. - **SSML pause tags:** `<silence time="1.5s" />` (interruptible) or `<hold time="2s" />` (not interruptible). `<silence>` lets the main agent jump in mid-pause; `<hold>` enforces dead air. - **Emotion markers:** `[laughter]`, `[sigh]`, etc. (can repeat like `[laughter] [laughter]`) - **Special actions:** `<endcall />` to end the call - Action tags only work in verbatim steps, not inside `<ai_generated>`. - **`then`** (array of strings, optional): Follow-up steps in order. Each fires on the actor's next turn after one main-agent reply, regardless of its content. `<interruption>` is only allowed here. **Usage Tips:** - Write `when` as third-person descriptions of what the main agent does (e.g. `"The agent confirms the cancellation"`) - End with an `<endcall />` step for clean call termination - Combine SSML tags and emotion markers naturally: `"Great <silence time=\"1.5s\" /> [laughter] [laughter]"`
  - `conditional_actions` ConditionalActions: Structured conditional-actions object for `scenario_type="conditional_actions"`. Use either `conditional_actions` (structured) or `instructions` (plain text) on a scenario — not both.
    - `role` string, required: Caller role / persona for this scenario. Example: `"You are a frustrated patient calling about a billing error"`.
    - `first_message` union: What the caller says first, always verbatim; empty means wait for the agent. Pass `{"say": "...", "then": [...]}` to add follow-up steps.
      - string
      - object
        - `say` string
        - `then` string[]
    - `conditions` Condition[], required: Ordered list of conditions and their actions.
      - `when` string, required: Third-person description of the agent's observable action that triggers this condition, matched semantically against the conversation.
      - `say` string, required: Spoken verbatim by the caller. Wrap the whole step in `<ai_generated>...</ai_generated>` to let the caller improvise from it instead; action tags such as `<endcall />` and `<audio id="..."/>` only work in verbatim steps.
      - `then` string[]: Optional follow-up steps, in order. Each fires on the caller's next turn after one agent reply, whatever the agent says. `<interruption>` is only allowed here.
    - `functions` object[]: Optional custom functions the caller can invoke mid-call from a step.
  - `expected_outcome_prompt` string: Expected Outcome Prompt Example: `"The user should be able to complete the order"`
  - `metrics` integer[]: List of metric IDs to associate with this evaluator Example: `[123, 456, 789]`
  - `tags` string[]: List of tags to associate with the evaluator Example: `["tag1", "tag2", "tag3"]`
  - `tool_ids` string[]: List of tool IDs to use for evaluator
  - `test_profile` integer: The test profile ID to use for the evaluator
  - `phone_number` integer: The phone number ID to use for the evaluator (works for both inbound and outbound)
  - `inbound_phone_number` integer: (Deprecated) The inbound phone number ID to use for the evaluator. Use phone_number instead, which works for both inbound and outbound.
  - `folder_path` string, nullable: Dot-separated path of the folder to assign this evaluator to (e.g. "Sales.Inbound"). Folders must be created before use — they are not auto-created. Use `scenarios_create_folder_create` to create each level before assigning evaluators. Example: to use "Sales.Inbound", first create "Sales", then create "Inbound" inside it.
  - `generated_mock_tool_entries` unknown
  - `first_message` string: Opening line the simulated caller says to the agent. Defaults to "Hello".
  - `max_duration` integer, nullable: Max call duration in seconds for this scenario. If not provided or set to null, the scenario will use the project's max_call_duration setting. Valid range: 10-3600 seconds (10 seconds to 60 minutes) Example: `600` (10 minutes)
  - `scenario_language` 'af' | 'ar' | 'bn' | 'bg' | 'zh' | 'cs' | 'da' | 'nl' | 'en' | 'et' | 'fi' | 'fr' | 'de' | 'el' | 'gu' | 'hi' | 'he' | 'hu' | 'id' | 'it' | 'ja' | 'kn' | 'kk' | 'ko' | 'ms' | 'ml' | 'mr' | 'multi' | 'no' | 'pl' | 'pa' | 'pt' | 'ro' | 'ru' | 'sk' | 'es' | 'sv' | 'th' | 'tr' | 'tl' | 'ta' | 'te' | 'uk' | 'uz' | 'vi': Language of the evaluator. Must match the chosen `personality`'s language — mismatches are rejected. * `af` - Afrikaans * `ar` - Arabic * `bn` - Bengali * `bg` - Bulgarian * `zh` - Chinese Simplified * `cs` - Czech * `da` - Danish * `nl` - Dutch * `en` - English * `et` - Estonian * `fi` - Finnish * `fr` - French * `de` - German * `el` - Greek * `gu` - Gujarati * `hi` - Hindi * `he` - Hebrew * `hu` - Hungarian * `id` - Indonesian * `it` - Italian * `ja` - Japanese * `kn` - Kannada * `kk` - Kazakh * `ko` - Korean * `ms` - Malay * `ml` - Malayalam * `mr` - Marathi * `multi` - Multilingual * `no` - Norwegian * `pl` - Polish * `pa` - Punjabi * `pt` - Portuguese * `ro` - Romanian * `ru` - Russian * `sk` - Slovak * `es` - Spanish * `sv` - Swedish * `th` - Thai * `tr` - Turkish * `tl` - Tagalog * `ta` - Tamil * `te` - Telugu * `uk` - Ukrainian * `uz` - Uzbek * `vi` - Vietnamese

## Response `201`

- SchemaScenario
  - `id` integer
  - `name` string: Name of the evaluator
  - `agent` integer: Agent ID Example: `2142`
  - `project` integer, nullable
  - `assistant_id` string: Alternative to agent ID - the assistant ID to use for this scenario Example: `"asst_1234567890"`
  - `personality` integer: ID of the personality
  - `personality_name` string: Name of the personality
  - `agent_name` string: Name of the agent associated with this scenario Example: `"Customer Support Agent"`
  - `tags` string[]: List of tags of the evaluators Example: `["tag1", "tag2", "tag3"]`
  - `tool_ids` unknown
  - `metrics` integer[]: List of metric IDs Example: `[123, 456, 789]`
  - `metric_names` string[]: List of metric names Example: `["Metric 1", "Metric 2", "Metric 3"]`
  - `phone_number` string: Phone number eg: +17776666333
  - `outbound_phone_number_data` PhoneNumberInbound
    - `id` integer
    - `number` string, required
    - `phone_number_id` string, required
  - `first_message` string: First message of the evaluator
  - `inbound_phone_number` integer, nullable
  - `inbound_phone_number_data` PhoneNumberInbound
    - `id` integer
    - `number` string, required
    - `phone_number_id` string, required
  - `instructions` string: Scenario Instructions - format depends on scenario_type: - For `instruction` type: Plain text instructions - For `conditional_actions` type: JSON string with `role`, `first_message` and a `conditions` array. Each condition has `when` (a natural-language third-person description of the main agent's observable action — e.g. `"The agent asks for your date of birth"`, matched semantically), `say` (spoken verbatim, with optional `<silence time="1.5s" />` (interruptible) or `<hold time="2s" />` (not interruptible) and `[laughter]` tags; wrap the whole step in `<ai_generated>...</ai_generated>` to improvise instead) and optional `then` follow-up steps. Responses return this format unless `conditional_actions_format=legacy` is passed. The earlier id-based format (`id`, `condition`, `action`, `type`, `fixed_message`) is still accepted on writes.
  - `conditional_actions` unknown
  - `condition_audio` object: Pre-recorded clips available to this scenario's conditional actions, keyed by clip id. Read-only: upload, replace and delete clips, and correct a transcript, through `condition-audio/`. Reference a clip from any number of verbatim steps by pasting its `tag`; `steps` lists where it is used (e.g. `conditions[1].then[0]`). Example: ```json { "greeting": {"tag": "<audio id=\"greeting\"/>", "filename": "greeting.wav", "status": "ready", "duration": 2.4, "transcript": "Hi, is this the clinic?", "steps": ["first_message", "conditions[1]"]} } ```
  - `simulation_description` string
  - `information_fields` object: Information fields of the evaluator Example: ```json { "user_name": "John Doe", "user_email": "john.doe@example.com", } ```
  - `expected_outcome_prompt` string: Expected outcome prompt of the evaluator Example: `"The user should be able to complete the order"`
  - `scenario_language` 'af' | 'ar' | 'bn' | 'bg' | 'zh' | 'cs' | 'da' | 'nl' | 'en' | 'et' | 'fi' | 'fr' | 'de' | 'el' | 'gu' | 'hi' | 'he' | 'hu' | 'id' | 'it' | 'ja' | 'kn' | 'kk' | 'ko' | 'ms' | 'ml' | 'mr' | 'multi' | 'no' | 'pl' | 'pa' | 'pt' | 'ro' | 'ru' | 'sk' | 'es' | 'sv' | 'th' | 'tr' | 'tl' | 'ta' | 'te' | 'uk' | 'uz' | 'vi': Language of the scenario * `af` - Afrikaans * `ar` - Arabic * `bn` - Bengali * `bg` - Bulgarian * `zh` - Chinese Simplified * `cs` - Czech * `da` - Danish * `nl` - Dutch * `en` - English * `et` - Estonian * `fi` - Finnish * `fr` - French * `de` - German * `el` - Greek * `gu` - Gujarati * `hi` - Hindi * `he` - Hebrew * `hu` - Hungarian * `id` - Indonesian * `it` - Italian * `ja` - Japanese * `kn` - Kannada * `kk` - Kazakh * `ko` - Korean * `ms` - Malay * `ml` - Malayalam * `mr` - Marathi * `multi` - Multilingual * `no` - Norwegian * `pl` - Polish * `pa` - Punjabi * `pt` - Portuguese * `ro` - Romanian * `ru` - Russian * `sk` - Slovak * `es` - Spanish * `sv` - Swedish * `th` - Thai * `tr` - Turkish * `tl` - Tagalog * `ta` - Tamil * `te` - Telugu * `uk` - Ukrainian * `uz` - Uzbek * `vi` - Vietnamese
  - `scenario_type` string: Type of scenario: - `instruction`: Standard instruction-based scenario (plain text instructions) - `conditional_actions`: Conditional action-based scenario (JSON with role, conditions, SSML tags, emotion markers)
  - `suite_type` 'undefined' | 'infrastructure': Type of predefined suite (e.g., infrastructure) * `undefined` - Undefined * `infrastructure` - Infrastructure
  - `is_predefined` boolean: Whether this is a predefined scenario (template)
  - `test_profile` integer, nullable
  - `test_profile_data` AIAgentTestProfile
    - `id` integer
    - `project` integer, nullable
    - `agent` integer, nullable
    - `agent_name` string: Name of the agent associated with this test profile Example: `"Customer Support Agent"`
    - `name` string, required
    - `information` unknown
    - `created_by` integer, nullable: ID of the user who created this test profile
    - `last_updated_by` integer, nullable: ID of the user who last updated this test profile
    - `created_at` string, date-time: Timestamp when the test profile was created
    - `updated_at` string, date-time: Timestamp when the test profile was last updated
  - `dynamic_variable_values` unknown
  - `generated_mock_tool_entries` unknown
  - `folder_path` string, nullable: Path of an existing folder to assign this evaluator to, e.g. "Sales" or "Sales.Inbound". Create it first with the create-folder endpoint.
  - `max_duration` integer, nullable: Max call duration in seconds for this scenario. If not provided or set to null, the scenario will use the project's max_call_duration setting. Valid range: 10-3600 seconds (10 seconds to 60 minutes) Example: `600` (10 minutes)

## Other responses

- `400`

## Changes

- **2026-11-03** `a6fb6a739212`: 1 info
  - endpoint added

[Operation history](https://skmtc.dev/cekura/apis/cekura-api/changes/test_framework/v2/scenarios/post.md)

---

[API](https://skmtc.dev/cekura/apis/cekura-api.md) - [All operations](https://skmtc.dev/cekura/apis/cekura-api/llms.txt) - [OpenAPI document](https://skmtc.dev/cekura/apis/cekura-api/revisions/a6fb6a739212?raw)