Sign in
Automation rule evaluators

Create automation rule evaluator

Create automation rule evaluator

post/v1/private/automations/evaluators

Request body

project_idstring uuid

Primary project ID (legacy field for backwards compatibility)

project_idsstring[]

Project IDs for write operations (used when creating/updating rules)

namestring required
sampling_ratenumber float

Fraction of production (SDK-logged) items this rule scores, from 0 to 1. Trace rules ignore this value for experiment, playground and optimization traces and score them in full; span and thread rules only ever evaluate SDK-logged data.

enabledboolean
trigger_scope'production' | 'experiment' | 'both'

Controls whether the rule fires on production traces, experiment traces, or both. Defaults to 'production' if omitted.

type'llm_as_judge' | 'user_defined_metric_python' | 'trace_thread_llm_as_judge' | 'trace_thread_user_defined_metric_python' | 'span_llm_as_judge' | 'span_user_defined_metric_python' required
action'evaluator' required

Response

Created

Changes

Changed in 3 of the 58 revisions of this API.52

  • 0739fa5022ca4See the full diff
    • ●

      the sampling_rate request property's exclusiveMaximum was set to false

      request-property-exclusive-max-set

    • ●

      the sampling_rate request property's exclusiveMinimum was set to false

      request-property-exclusive-min-set

    • ●

      the sampling_rate request property's max was set to 1.00

      request-property-max-set

    • ●

      the sampling_rate request property's min was set to 0.00

      request-property-min-set

  • 0bfce600211c11See the full diff
    • ●

      the name request property's maxLength was set to 150

      request-property-max-length-set

    • ○

      the name request property's minLength was decreased from 1 to 0

      request-property-min-length-decreased

  • 2ad1c641e6571See the full diff
    • ○

      added the new optional request property trigger_scope

      new-optional-request-property