Update A/B test

Percentage of the agent's traffic routed to the variant. Running tests for the same agent cannot exceed 100 in total.

exploratory compares descriptive metrics; confidence follows a locked statistical plan with planned decision looks.

Completed runs needed in each group before the comparison reports enough data.

Pause the test automatically when the variant's failure rate exceeds rollback_threshold.

Variant failure rate (0 to 1) above which the failure guardrail pauses the test.

Number of most recent finished variant runs the failure guardrail evaluates.

Pause the test when the variant's average judge score falls below quality_threshold.

Judge for the quality guardrail. It must belong to the same agent and environment, be enabled and automatic, evaluate every run and have no filter.

Lowest acceptable average normalized score (0 to 1) for the variant.

Number of most recent evaluated variant runs the quality guardrail averages.

Evaluations needed before the quality guardrail can act. Cannot exceed quality_window.

running starts a draft or resumes a paused test, paused pauses a running test, and concluded ends a running or paused test.

Manually chosen winner. Not accepted for confidence tests.

Project the test belongs to.

Environment the test runs in.

Base (control) agent.

Variant suffix.

Variant agent name, {agent_name}-test-{test_name}.

Percentage of the agent's traffic routed to the variant.

A/B test state: draft until started, running while routing traffic, paused, concluded, or failed when the variant is missing from the active deployment. rolled_back is no longer set.

exploratory compares descriptive metrics; confidence follows a locked statistical plan with planned decision looks.

Completed runs needed in each group before the comparison reports enough data.

Whether the failure guardrail is on.

Variant failure rate (0 to 1) above which the failure guardrail pauses the test.

Number of most recent finished variant runs the failure guardrail evaluates.

Whether the quality guardrail is on.

Judge for the quality guardrail. null when unset or without judges.view.

Lowest acceptable average normalized score (0 to 1) for the variant.

Number of most recent evaluated variant runs the quality guardrail averages.

Evaluations needed before the quality guardrail can act.

Manually chosen winner of an exploratory test. null when none.

Test verdict: collecting before the first planned look, continue after a look without a decision, variant_recommended or control_recommended when a boundary was crossed, inconclusive, invalid_srm after a sample-ratio mismatch, or safety_stopped when the test was stopped automatically.

Explanation of recommendation.

Why the test was stopped automatically. Cleared when the test resumes.

Why a test stopped: manual, a guardrail (failure_guardrail, quality_guardrail), a planned look (efficacy when a boundary was crossed, futility when the final look was inconclusive), srm (sample-ratio mismatch), analysis_error (metric judge evaluations failed), or variant_unavailable.

When the test first started. null for drafts.

When the test was last paused or stopped. Cleared when it resumes.

When the test was concluded.

Completed control runs. Confidence tests exclude forced runs.

Completed variant runs. Confidence tests exclude forced runs.

Always null. Cost statistics are returned by the comparison endpoint.

Always null. Cost statistics are returned by the comparison endpoint.

Average duration of completed control runs, in milliseconds.

Average duration of completed variant runs, in milliseconds.

Average normalized judge score (0 to 1) of control runs, from the metric or quality judge when set. null without evaluations.

Average normalized judge score (0 to 1) of variant runs, from the metric or quality judge when set. null without evaluations.