AI Tests

Bulk create test metrics

Changed on

Creates several metrics in one request. Accepts an array of metric objects and returns the created metrics.

post/ai/tests/test_framework/v1/metrics/bulk_create

Request

  • Base URL: https://api.telnyx.com/v2
  • URL: https://api.telnyx.com/v2/ai/tests/test_framework/v1/metrics/bulk_create
  • Auth: HTTP bearer

Request body

namestring required
descriptionstring
typestring

How the metric is evaluated: custom_code or llm_judge.

eval_typestring

What the metric scores: numeric, continuous_qualitative, or binary_workflow_adherence.

promptstring

Judge prompt used by llm_judge metrics.

enum_valuesstring[]
agentsinteger[]
assistant_idstring
add_to_new_agentsboolean nullable
simulation_enabledboolean
observability_enabledboolean
sampling_enabledboolean
audio_enabledboolean
display_orderinteger

Example request

[
  {
    "name": "Greets within 5 seconds",
    "description": "Returns whether the assistant greeted the caller within five seconds.",
    "type": "llm_judge",
    "eval_type": "continuous_qualitative",
    "prompt": "Score whether the greeting was timely and natural."
  }
]

Response

Test metrics created.

idinteger required
slugstring nullable

Stable identifier of a built-in metric.

projectinteger nullable
agentinteger nullable
agentsinteger[]

Test agent IDs the metric is attached to.

agent_namesstring[]
namestring required
descriptionstring
function_namestring nullable

Built-in evaluation function used by custom_code metrics.

typestring required

How the metric is evaluated: custom_code or llm_judge.

eval_typestring required

What the metric scores: numeric, continuous_qualitative, or binary_workflow_adherence.

enum_valuesstring[]

Allowed values for qualitative enum metrics.

audio_enabledboolean
observability_enabledboolean
simulation_enabledboolean
sampling_enabledboolean
sample_ratenumber nullable
promptstring

Judge prompt used by llm_judge metrics.

display_orderinteger
evaluation_triggerstring
trigger_typestring
evaluation_trigger_promptstring
evaluation_trigger_custom_codestring
priority_assignment_promptstring
configurationobject nullable
kb_file_idsinteger[]
knowledge_base_filesstring[]
metric_costnumber
alert_typestring
significant_change_alert_statusstring
significant_change_alert_directionstring
window_sizeinteger

Sliding window size the alert status is computed over.

std_multipliernumber

Standard-deviation multiplier for significant-change alerts.

ewma_alphanumber

Smoothing factor for the alert trend.

add_to_new_agentsboolean nullable

Whether the metric is attached to newly created test agents by default.

alert_filtersobject nullable
uses_llm_judgeboolean
user_definedboolean

Example response

[
  {
    "id": 266881,
    "slug": "ai_interrupting_user",
    "project": 16003,
    "agent": null,
    "agents": [],
    "agent_names": [],
    "name": "AI interrupting user",
    "description": "Tells if the AI (main agent) interrupted user (testing agent) during the interaction.",
    "function_name": "get_main_agent_interruption",
    "type": "custom_code",
    "eval_type": "numeric",
    "enum_values": [],
    "audio_enabled": false,
    "observability_enabled": true,
    "simulation_enabled": true,
    "sampling_enabled": false,
    "sample_rate": null,
    "prompt": "",
    "display_order": 1,
    "evaluation_trigger": "always",
    "trigger_type": "llm_judge",
    "evaluation_trigger_prompt": "",
    "evaluation_trigger_custom_code": "",
    "priority_assignment_prompt": "",
    "configuration": {},
    "overall_score": 0,
    "total_score": 0,
    "kb_file_ids": [],
    "knowledge_base_files": [],
    "metric_cost": 0,
    "alert_type": "disabled",
    "significant_change_alert_status": "disabled",
    "significant_change_alert_direction": "",
    "window_size": 50,
    "std_multiplier": 2,
    "ewma_alpha": 0.1,
    "add_to_new_agents": true,
    "alert_filters": {},
    "uses_llm_judge": false,
    "user_defined": false
  }
]

Changes