run

Validate Rule

Changed on

Validate a rule by executing it with test data without creating a saved rule.

This endpoint allows testing LLM-as-judge evaluators before saving them. It accepts a rule configuration (same as rule creation) and test data, executes the evaluator, and returns the evaluation results in the same format as batch_invoke_evaluator.

Only LLM-as-judge rules (evaluators) are supported. Code evaluators are not allowed.

The evaluator execution traces are written to the database (in the "evaluators" project), which allows users to see the evaluator execution history.

post/api/v1/runs/rules/validate

Request

  • Base URL
    https://api.smith.langchain.com
  • URL
    https://api.smith.langchain.com/api/v1/runs/rules/validate
  • Auth
    • API key in header X-API-Key
    • or API key in header X-Tenant-Id
    • or HTTP bearer

Request body

display_namestring required
session_idstring uuid nullable
is_enabledboolean
dataset_idstring uuid nullable
sampling_ratenumber required
filterstring nullable
trace_filterstring nullable
tree_filterstring nullable
backfill_fromstring date-time nullable
use_corrections_datasetboolean
num_few_shot_examplesinteger nullable
extend_onlyboolean
is_tracing_disabledboolean
extend_evaluator_trace_retentionboolean
extend_dataset_trace_retentionboolean nullable
extend_annotation_queue_trace_retentionboolean nullable
extend_webhook_trace_retentionboolean nullable
transientboolean
add_to_annotation_queue_idstring uuid nullable
add_to_dataset_idstring uuid nullable
add_to_dataset_prefer_correctionboolean
evaluator_idstring uuid nullable
evaluator_versioninteger nullable
create_alignment_queueboolean
include_extended_statsboolean
group_by'thread_id' | 'trajectory_id' nullable
tracer_session_issue_idstring uuid nullable
test_inputsobject nullable
test_outputsobject nullable
test_reference_outputsobject nullable
test_attachmentsobject nullable
test_thread_idstring nullable
validate_project_compatibilityboolean

Response

Successful Response

object[] required

Changes

  • ec156df8da3aRevision changes
    • info

      added the new trajectory enum value to the request property ////

  • 1a8ee9358d81Revision changes
    • breaking

      request property / was restricted to a list of enum values

    • breaking

      added the pattern ^(?!^[-+.]*$)[+-]?0*\d*\.?\d*$ to the request property ///

    • breaking

      the / response's property type changed from object to string for status

    • breaking

      removed the required property // from the response with the status

    • breaking

      removed the required property // from the response with the status

    • breaking

      removed the required property // from the response with the status

    • info

      added the new optional request property ///

    • info

      added the new optional request property ///

    • info

      added the new optional request property ///

    • info

      added the new optional request property ////

    • info

      added the new optional request property

    • info

      added the new optional request property

    • info

      added the new optional request property

    • info

      added the new optional request property

    • info

      added the new optional request property

    • info

      added the new optional request property

    • info

      added the new optional request property

    • info

      the request property const value thread_id was removed

    • info

      added the new thread_id enum value to the request property /

    • info

      added the new trajectory_id enum value to the request property /