AI Tests

Create a test metric

Changed on

Creates a metric in your organization.

post/ai/tests/test_framework/v1/metrics

Request

  • Base URL: https://api.telnyx.com/v2
  • URL: https://api.telnyx.com/v2/ai/tests/test_framework/v1/metrics
  • Auth: HTTP bearer

Request body

namestring required
descriptionstring
typestring

How the metric is evaluated: custom_code or llm_judge.

eval_typestring

What the metric scores: numeric, continuous_qualitative, or binary_workflow_adherence.

promptstring

Judge prompt used by llm_judge metrics.

enum_valuesstring[]
agentsinteger[]
assistant_idstring
add_to_new_agentsboolean nullable
simulation_enabledboolean
observability_enabledboolean
sampling_enabledboolean
audio_enabledboolean
display_orderinteger

Example request

{
  "name": "Escalates on refund denial",
  "description": "Returns a score for whether the assistant escalates when the caller's refund is denied.",
  "type": "llm_judge",
  "eval_type": "continuous_qualitative",
  "prompt": "Score the assistant's escalation behaviour between 0.0 and 1.0.",
  "agents": [
    1259
  ],
  "add_to_new_agents": false,
  "simulation_enabled": true,
  "observability_enabled": true
}

Response

Test metric created.

idinteger required
slugstring nullable

Stable identifier of a built-in metric.

projectinteger nullable
agentinteger nullable
agentsinteger[]

Test agent IDs the metric is attached to.

agent_namesstring[]
namestring required
descriptionstring
function_namestring nullable

Built-in evaluation function used by custom_code metrics.

typestring required

How the metric is evaluated: custom_code or llm_judge.

eval_typestring required

What the metric scores: numeric, continuous_qualitative, or binary_workflow_adherence.

enum_valuesstring[]

Allowed values for qualitative enum metrics.

audio_enabledboolean
observability_enabledboolean
simulation_enabledboolean
sampling_enabledboolean
sample_ratenumber nullable
promptstring

Judge prompt used by llm_judge metrics.

display_orderinteger
evaluation_triggerstring
trigger_typestring
evaluation_trigger_promptstring
evaluation_trigger_custom_codestring
priority_assignment_promptstring
configurationobject nullable
kb_file_idsinteger[]
knowledge_base_filesstring[]
metric_costnumber
alert_typestring
significant_change_alert_statusstring
significant_change_alert_directionstring
window_sizeinteger

Sliding window size the alert status is computed over.

std_multipliernumber

Standard-deviation multiplier for significant-change alerts.

ewma_alphanumber

Smoothing factor for the alert trend.

add_to_new_agentsboolean nullable

Whether the metric is attached to newly created test agents by default.

alert_filtersobject nullable
uses_llm_judgeboolean
user_definedboolean

Example response

{
  "id": 266881,
  "slug": "ai_interrupting_user",
  "project": 16003,
  "agent": null,
  "agents": [],
  "agent_names": [],
  "name": "AI interrupting user",
  "description": "Tells if the AI (main agent) interrupted user (testing agent) during the interaction.",
  "function_name": "get_main_agent_interruption",
  "type": "custom_code",
  "eval_type": "numeric",
  "enum_values": [],
  "audio_enabled": false,
  "observability_enabled": true,
  "simulation_enabled": true,
  "sampling_enabled": false,
  "sample_rate": null,
  "prompt": "",
  "display_order": 1,
  "evaluation_trigger": "always",
  "trigger_type": "llm_judge",
  "evaluation_trigger_prompt": "",
  "evaluation_trigger_custom_code": "",
  "priority_assignment_prompt": "",
  "configuration": {},
  "overall_score": 0,
  "total_score": 0,
  "kb_file_ids": [],
  "knowledge_base_files": [],
  "metric_cost": 0,
  "alert_type": "disabled",
  "significant_change_alert_status": "disabled",
  "significant_change_alert_direction": "",
  "window_size": 50,
  "std_multiplier": 2,
  "ewma_alpha": 0.1,
  "add_to_new_agents": true,
  "alert_filters": {},
  "uses_llm_judge": false,
  "user_defined": false
}

Changes