Sign in
run

Validate Rule

Validate a rule by executing it with test data without creating a saved rule.

This endpoint allows testing LLM-as-judge evaluators before saving them. It accepts a rule configuration (same as rule creation) and test data, executes the evaluator, and returns the evaluation results in the same format as batch_invoke_evaluator.

Only LLM-as-judge rules (evaluators) are supported. Code evaluators are not allowed.

The evaluator execution traces are written to the database (in the "evaluators" project), which allows users to see the evaluator execution history.

post/api/v1/runs/rules/validate

Request body

display_namestring required
session_idstring uuid nullable
is_enabledboolean
dataset_idstring uuid nullable
sampling_ratenumber required
filterstring nullable
trace_filterstring nullable
tree_filterstring nullable
backfill_fromstring date-time nullable
use_corrections_datasetboolean
num_few_shot_examplesinteger nullable
extend_onlyboolean
transientboolean
add_to_annotation_queue_idstring uuid nullable
add_to_dataset_idstring uuid nullable
add_to_dataset_prefer_correctionboolean
evaluator_idstring uuid nullable
evaluator_versioninteger nullable
create_alignment_queueboolean
include_extended_statsboolean
group_by'thread_id' nullable
tracer_session_issue_idstring uuid nullable
test_inputsobject nullable
test_outputsobject nullable
test_reference_outputsobject nullable
test_attachmentsobject nullable

Response

Successful Response

object[] required

Changes

Changed in 2 of the 48 revisions of this API.13

  • a0acb3a6a10112See the full diff
    • ▲

      removed RunRuleSpendLimitSchema from the spend_limit request property anyOf list

      request-property-any-of-removed

    • ○

      added the new optional request property tracer_session_issue_id

      new-optional-request-property

    • ○

      added RunRuleSpendLimitSchema-Input to the spend_limit request property anyOf list

      request-property-any-of-added

    This revision also has 6 changes that name no endpoint, such as unreferenced schemas being removed. See the revision's changelog

  • 0480571f80d71See the full diff
    • ○

      added the new optional request property spend_limit

      new-optional-request-property