--- title: "Update A/B test" method: PATCH path: "/v1/projects/{project_id}/ab-tests/{test_id}" tags: ["ab-tests"] --- # Update A/B test `PATCH /v1/projects/{project_id}/ab-tests/{test_id}` ## Request - Base URL: `https://api.connic.co` - URL: `https://api.connic.co/v1/projects/{project_id}/ab-tests/{test_id}` - Auth: HTTP bearer Update an A/B test's configuration, lifecycle state, or outcome. Set `status` to `running` to start or resume, `paused` to pause, or `concluded` to end the test. Analysis mode and plan can only change while the test is a draft, and a confidence test's configuration is fixed once it starts. Required permission: `ab_tests.update`. With `judges.view`, the response also includes judge names. ## Path parameters - `project_id` string, uuid, required - `test_id` string, uuid, required ## Request body - ABTestUpdate: Partial update of an A/B test. Mode and plan change only in draft; confidence settings lock at the first start. - `traffic_percentage` integer, nullable: Percentage of the agent's traffic routed to the variant. Running tests for the same agent cannot exceed 100 in total. - `analysis_mode` 'exploratory' | 'confidence': `exploratory` compares descriptive metrics; `confidence` follows a locked statistical plan with planned decision looks. - `analysis_plan` ABTestAnalysisPlanInput: Assumptions for a confidence-mode A/B test plan. Metric values use a 0 to 1 scale. - `primary_metric` 'success_rate' | 'judge_score', required: `success_rate` compares the share of completed runs; `judge_score` compares the normalized score from `judge_id`. - `objective` 'superiority' | 'noninferiority': `superiority` tests whether the variant beats control; `noninferiority` tests that it is not worse by more than a margin. - `direction` 'increase' | 'decrease': Which change in the primary metric counts as better: `increase` or `decrease`. - `baseline_value` number, required: Expected control value: a success rate between 0 and 1 (exclusive), or a judge score from 0 to 1. - `baseline_stddev` number, nullable: Expected standard deviation of the normalized judge score. Required and positive for `judge_score`. - `minimum_detectable_effect` number, required: Smallest absolute difference worth detecting, for example 0.05 for 5 percentage points. - `noninferiority_margin` number, nullable: Largest acceptable absolute shortfall versus control. Required for `noninferiority`. - `alpha` number: Two-sided significance level. - `power` number: Probability of detecting the planned effect. - `randomization_unit` 'run' | 'session': `run` assigns each run independently; `session` keeps all runs of one session in the same arm. - `judge_id` string, uuid, nullable: Judge that scores runs for `judge_score`. It must belong to the same agent and environment, be enabled and automatic, evaluate every run and have no filter. Must be null for `success_rate`. - `futility_threshold` number: Conditional power below which a look is flagged as unlikely to succeed. Advisory only; it does not stop the test. - `min_sample_size` integer, nullable: Completed runs needed in each group before the comparison reports enough data. - `auto_rollback_enabled` boolean, nullable: Pause the test automatically when the variant's failure rate exceeds `rollback_threshold`. - `rollback_threshold` number, nullable: Variant failure rate (0 to 1) above which the failure guardrail pauses the test. - `rollback_window` integer, nullable: Number of most recent finished variant runs the failure guardrail evaluates. - `quality_guardrail_enabled` boolean, nullable: Pause the test when the variant's average judge score falls below `quality_threshold`. - `quality_judge_id` string, uuid, nullable: Judge for the quality guardrail. It must belong to the same agent and environment, be enabled and automatic, evaluate every run and have no filter. - `quality_threshold` number, nullable: Lowest acceptable average normalized score (0 to 1) for the variant. - `quality_window` integer, nullable: Number of most recent evaluated variant runs the quality guardrail averages. - `quality_min_evaluations` integer, nullable: Evaluations needed before the quality guardrail can act. Cannot exceed `quality_window`. - `status` 'running' | 'paused' | 'concluded', nullable: `running` starts a draft or resumes a paused test, `paused` pauses a running test, and `concluded` ends a running or paused test. - `winner` 'control' | 'variant', nullable: Manually chosen winner. Not accepted for confidence tests. ## Response `200` Successful Response - ABTestResponse: An A/B test that routes part of an agent's traffic to a variant, with run totals per arm. - `id` string, uuid, required - `project_id` string, uuid, required: Project the test belongs to. - `environment_id` string, uuid, required: Environment the test runs in. - `agent_name` string, required: Base (control) agent. - `test_name` string, required: Variant suffix. - `variant_agent_name` string, required: Variant agent name, `{agent_name}-test-{test_name}`. - `traffic_percentage` integer, required: Percentage of the agent's traffic routed to the variant. - `status` 'draft' | 'running' | 'paused' | 'concluded' | 'rolled_back' | 'failed', required: A/B test state: `draft` until started, `running` while routing traffic, `paused`, `concluded`, or `failed` when the variant is missing from the active deployment. `rolled_back` is no longer set. - `analysis_mode` 'exploratory' | 'confidence': `exploratory` compares descriptive metrics; `confidence` follows a locked statistical plan with planned decision looks. - `analysis_plan` ABTestAnalysisPlanResponse: Statistical plan of a confidence-mode A/B test, locked when the test first starts. - `primary_metric` 'success_rate' | 'judge_score', required: `success_rate` compares the share of completed runs; `judge_score` compares the normalized score from `judge_id`. - `objective` 'superiority' | 'noninferiority': `superiority` tests whether the variant beats control; `noninferiority` tests that it is not worse by more than a margin. - `direction` 'increase' | 'decrease': Which change in the primary metric counts as better: `increase` or `decrease`. - `baseline_value` number, required: Expected control value: a success rate between 0 and 1 (exclusive), or a judge score from 0 to 1. - `baseline_stddev` number, nullable: Expected standard deviation of the normalized judge score. Required and positive for `judge_score`. - `minimum_detectable_effect` number, required: Smallest absolute difference worth detecting, for example 0.05 for 5 percentage points. - `noninferiority_margin` number, nullable: Largest acceptable absolute shortfall versus control. Required for `noninferiority`. - `alpha` number: Two-sided significance level. - `power` number: Probability of detecting the planned effect. - `randomization_unit` 'run' | 'session': `run` assigns each run independently; `session` keeps all runs of one session in the same arm. - `judge_id` string, uuid, nullable: Judge that scores runs for `judge_score`. It must belong to the same agent and environment, be enabled and automatic, evaluate every run and have no filter. Must be null for `success_rate`. - `futility_threshold` number: Conditional power below which a look is flagged as unlikely to succeed. Advisory only; it does not stop the test. - `id` string, uuid, required - `ab_test_id` string, uuid, required: A/B test the plan belongs to. - `control_allocation` number, required: Share of assigned traffic that goes to control. - `variant_allocation` number, required: Share of assigned traffic that goes to the variant. - `judge_snapshot` object: Metric judge configuration captured at the first start. Empty for `success_rate` plans or before the start; only `redacted: true` without `judges.view`. - `quality_judge_snapshot` object: Quality guardrail judge configuration captured at the first start. Empty when the guardrail is off; only `redacted: true` without `judges.view`. - `planned_control_samples` integer, required: Maximum number of control units the plan collects. - `planned_variant_samples` integer, required: Maximum number of variant units the plan collects. - `look_fractions` number[], required: Shares of the planned sample at which analysis looks happen. - `boundaries` object[], required: Decision boundary for each planned look, in the same shape as the plan preview `boundaries`. - `design_version` string, required: Identifier of the sequential design, such as `two-sided-obf-v1`. - `srm_threshold` number, required: p-value below which a sample-ratio mismatch between the planned and observed split stops the test. - `outcome_policy` string, required: Identifier of the rule that turns each assigned unit into an outcome. - `locked_at` string, date-time, nullable: When the plan was locked by the test's first start. null while the test is a draft. - `created_at` string, date-time, required - `updated_at` string, date-time, required - `min_sample_size` integer, required: Completed runs needed in each group before the comparison reports enough data. - `auto_rollback_enabled` boolean, required: Whether the failure guardrail is on. - `rollback_threshold` number, required: Variant failure rate (0 to 1) above which the failure guardrail pauses the test. - `rollback_window` integer, required: Number of most recent finished variant runs the failure guardrail evaluates. - `quality_guardrail_enabled` boolean: Whether the quality guardrail is on. - `quality_judge_id` string, uuid, nullable: Judge for the quality guardrail. null when unset or without `judges.view`. - `quality_threshold` number: Lowest acceptable average normalized score (0 to 1) for the variant. - `quality_window` integer: Number of most recent evaluated variant runs the quality guardrail averages. - `quality_min_evaluations` integer: Evaluations needed before the quality guardrail can act. - `winner` 'control' | 'variant', nullable: Manually chosen winner of an exploratory test. null when none. - `recommendation` 'collecting' | 'continue' | 'variant_recommended' | 'control_recommended' | 'inconclusive' | 'invalid_srm' | 'safety_stopped': Test verdict: `collecting` before the first planned look, `continue` after a look without a decision, `variant_recommended` or `control_recommended` when a boundary was crossed, `inconclusive`, `invalid_srm` after a sample-ratio mismatch, or `safety_stopped` when the test was stopped automatically. - `recommendation_reason` string, nullable: Explanation of `recommendation`. - `error` string, nullable: Why the test was stopped automatically. Cleared when the test resumes. - `stop_reason` 'manual' | 'failure_guardrail' | 'quality_guardrail' | 'efficacy' | 'futility' | 'srm' | 'analysis_error' | 'variant_unavailable': Why a test stopped: `manual`, a guardrail (`failure_guardrail`, `quality_guardrail`), a planned look (`efficacy` when a boundary was crossed, `futility` when the final look was inconclusive), `srm` (sample-ratio mismatch), `analysis_error` (metric judge evaluations failed), or `variant_unavailable`. - `stop_details` object: Details for `stop_reason`, such as the observed failure rate and window for `failure_guardrail`. - `created_at` string, date-time, required - `updated_at` string, date-time, required - `started_at` string, date-time, nullable: When the test first started. null for drafts. - `stopped_at` string, date-time, nullable: When the test was last paused or stopped. Cleared when it resumes. - `concluded_at` string, date-time, nullable: When the test was concluded. - `control_runs` integer: Completed control runs. Confidence tests exclude forced runs. - `variant_runs` integer: Completed variant runs. Confidence tests exclude forced runs. - `control_avg_cost` number, nullable: Always null. Cost statistics are returned by the comparison endpoint. - `variant_avg_cost` number, nullable: Always null. Cost statistics are returned by the comparison endpoint. - `control_avg_duration` number, nullable: Average duration of completed control runs, in milliseconds. - `variant_avg_duration` number, nullable: Average duration of completed variant runs, in milliseconds. - `control_avg_score` number, nullable: Average normalized judge score (0 to 1) of control runs, from the metric or quality judge when set. null without evaluations. - `variant_avg_score` number, nullable: Average normalized judge score (0 to 1) of variant runs, from the metric or quality judge when set. null without evaluations. ## Other responses - `401`: Missing or invalid API key. - `403`: The API key lacks the required permission or access to the environment. - `404`: The resource does not exist in this project. - `422`: Validation Error - `429`: Rate limit exceeded. --- [API](https://skmtc.dev/connic/apis/connic-api.md) · [All operations](https://skmtc.dev/connic/apis/connic-api/llms.txt) · [OpenAPI document](https://skmtc.dev/connic/apis/connic-api/revisions/482932da5fbe?raw)