Automation rule evaluators

Create automation rule evaluator

Changed on

Create automation rule evaluator

post/v1/private/automations/evaluators

Request

  • Base URL
    http://localhost:5173/api
  • URL
    http://localhost:5173/api/v1/private/automations/evaluators
  • Auth
    none declared

Request body

project_idstring uuid

Primary project ID (legacy field for backwards compatibility)

project_idsstring[]

Project IDs for write operations (used when creating/updating rules)

namestring required
sampling_ratenumber float

Fraction of production (SDK-logged) items this rule scores, from 0 to 1. Trace rules ignore this value for experiment, playground and optimization traces and score them in full; span and thread rules only ever evaluate SDK-logged data.

enabledboolean
trigger_scope'production' | 'experiment' | 'both'

Controls whether the rule fires on production traces, experiment traces, or both. Defaults to 'production' if omitted.

type'llm_as_judge' | 'user_defined_metric_python' | 'trace_thread_llm_as_judge' | 'trace_thread_user_defined_metric_python' | 'span_llm_as_judge' | 'span_user_defined_metric_python' required
action'evaluator' | 'annotation_queue_router' required

Response

Created

Changes

  • 4baa2ed0236fRevision changes
    • info

      added the new annotation_queue_router enum value to the request property

  • 0739fa5022caRevision changes
    • warning

      the request property's exclusiveMaximum was set to false

    • warning

      the request property's exclusiveMinimum was set to false

    • warning

      the request property's max was set to 1.00

    • warning

      the request property's min was set to 0.00