--- title: "Get A/B test comparison" method: GET path: "/v1/projects/{project_id}/ab-tests/{test_id}/comparison" tags: ["ab-tests"] --- # Get A/B test comparison `GET /v1/projects/{project_id}/ab-tests/{test_id}/comparison` ## Request - Base URL: `https://api.connic.co` - URL: `https://api.connic.co/v1/projects/{project_id}/ab-tests/{test_id}/comparison` - Auth: HTTP bearer Compare control and variant performance for an A/B test. Required permission: `ab_tests.view`. With `judges.view`, the response also includes judge names. ## Path parameters - `project_id` string, uuid, required - `test_id` string, uuid, required ## Query parameters - `convert_costs_to` 'EUR' | 'USD', nullable: Convert token costs to EUR or USD at the daily ECB reference rate of each model call ## Response `200` Successful Response - PublicABTestComparisonResponse - `test` ABTestResponse, required: An A/B test that routes part of an agent's traffic to a variant, with run totals per arm. - `id` string, uuid, required - `project_id` string, uuid, required: Project the test belongs to. - `environment_id` string, uuid, required: Environment the test runs in. - `agent_name` string, required: Base (control) agent. - `test_name` string, required: Variant suffix. - `variant_agent_name` string, required: Variant agent name, `{agent_name}-test-{test_name}`. - `traffic_percentage` integer, required: Percentage of the agent's traffic routed to the variant. - `status` 'draft' | 'running' | 'paused' | 'concluded' | 'rolled_back' | 'failed', required: A/B test state: `draft` until started, `running` while routing traffic, `paused`, `concluded`, or `failed` when the variant is missing from the active deployment. `rolled_back` is no longer set. - `analysis_mode` 'exploratory' | 'confidence': `exploratory` compares descriptive metrics; `confidence` follows a locked statistical plan with planned decision looks. - `analysis_plan` ABTestAnalysisPlanResponse: Statistical plan of a confidence-mode A/B test, locked when the test first starts. - `primary_metric` 'success_rate' | 'judge_score', required: `success_rate` compares the share of completed runs; `judge_score` compares the normalized score from `judge_id`. - `objective` 'superiority' | 'noninferiority': `superiority` tests whether the variant beats control; `noninferiority` tests that it is not worse by more than a margin. - `direction` 'increase' | 'decrease': Which change in the primary metric counts as better: `increase` or `decrease`. - `baseline_value` number, required: Expected control value: a success rate between 0 and 1 (exclusive), or a judge score from 0 to 1. - `baseline_stddev` number, nullable: Expected standard deviation of the normalized judge score. Required and positive for `judge_score`. - `minimum_detectable_effect` number, required: Smallest absolute difference worth detecting, for example 0.05 for 5 percentage points. - `noninferiority_margin` number, nullable: Largest acceptable absolute shortfall versus control. Required for `noninferiority`. - `alpha` number: Two-sided significance level. - `power` number: Probability of detecting the planned effect. - `randomization_unit` 'run' | 'session': `run` assigns each run independently; `session` keeps all runs of one session in the same arm. - `judge_id` string, uuid, nullable: Judge that scores runs for `judge_score`. It must belong to the same agent and environment, be enabled and automatic, evaluate every run and have no filter. Must be null for `success_rate`. - `futility_threshold` number: Conditional power below which a look is flagged as unlikely to succeed. Advisory only; it does not stop the test. - `id` string, uuid, required - `ab_test_id` string, uuid, required: A/B test the plan belongs to. - `control_allocation` number, required: Share of assigned traffic that goes to control. - `variant_allocation` number, required: Share of assigned traffic that goes to the variant. - `judge_snapshot` object: Metric judge configuration captured at the first start. Empty for `success_rate` plans or before the start; only `redacted: true` without `judges.view`. - `quality_judge_snapshot` object: Quality guardrail judge configuration captured at the first start. Empty when the guardrail is off; only `redacted: true` without `judges.view`. - `planned_control_samples` integer, required: Maximum number of control units the plan collects. - `planned_variant_samples` integer, required: Maximum number of variant units the plan collects. - `look_fractions` number[], required: Shares of the planned sample at which analysis looks happen. - `boundaries` object[], required: Decision boundary for each planned look, in the same shape as the plan preview `boundaries`. - `design_version` string, required: Identifier of the sequential design, such as `two-sided-obf-v1`. - `srm_threshold` number, required: p-value below which a sample-ratio mismatch between the planned and observed split stops the test. - `outcome_policy` string, required: Identifier of the rule that turns each assigned unit into an outcome. - `locked_at` string, date-time, nullable: When the plan was locked by the test's first start. null while the test is a draft. - `created_at` string, date-time, required - `updated_at` string, date-time, required - `min_sample_size` integer, required: Completed runs needed in each group before the comparison reports enough data. - `auto_rollback_enabled` boolean, required: Whether the failure guardrail is on. - `rollback_threshold` number, required: Variant failure rate (0 to 1) above which the failure guardrail pauses the test. - `rollback_window` integer, required: Number of most recent finished variant runs the failure guardrail evaluates. - `quality_guardrail_enabled` boolean: Whether the quality guardrail is on. - `quality_judge_id` string, uuid, nullable: Judge for the quality guardrail. null when unset or without `judges.view`. - `quality_threshold` number: Lowest acceptable average normalized score (0 to 1) for the variant. - `quality_window` integer: Number of most recent evaluated variant runs the quality guardrail averages. - `quality_min_evaluations` integer: Evaluations needed before the quality guardrail can act. - `winner` 'control' | 'variant', nullable: Manually chosen winner of an exploratory test. null when none. - `recommendation` 'collecting' | 'continue' | 'variant_recommended' | 'control_recommended' | 'inconclusive' | 'invalid_srm' | 'safety_stopped': Test verdict: `collecting` before the first planned look, `continue` after a look without a decision, `variant_recommended` or `control_recommended` when a boundary was crossed, `inconclusive`, `invalid_srm` after a sample-ratio mismatch, or `safety_stopped` when the test was stopped automatically. - `recommendation_reason` string, nullable: Explanation of `recommendation`. - `error` string, nullable: Why the test was stopped automatically. Cleared when the test resumes. - `stop_reason` 'manual' | 'failure_guardrail' | 'quality_guardrail' | 'efficacy' | 'futility' | 'srm' | 'analysis_error' | 'variant_unavailable': Why a test stopped: `manual`, a guardrail (`failure_guardrail`, `quality_guardrail`), a planned look (`efficacy` when a boundary was crossed, `futility` when the final look was inconclusive), `srm` (sample-ratio mismatch), `analysis_error` (metric judge evaluations failed), or `variant_unavailable`. - `stop_details` object: Details for `stop_reason`, such as the observed failure rate and window for `failure_guardrail`. - `created_at` string, date-time, required - `updated_at` string, date-time, required - `started_at` string, date-time, nullable: When the test first started. null for drafts. - `stopped_at` string, date-time, nullable: When the test was last paused or stopped. Cleared when it resumes. - `concluded_at` string, date-time, nullable: When the test was concluded. - `control_runs` integer: Completed control runs. Confidence tests exclude forced runs. - `variant_runs` integer: Completed variant runs. Confidence tests exclude forced runs. - `control_avg_cost` number, nullable: Always null. Cost statistics are returned by the comparison endpoint. - `variant_avg_cost` number, nullable: Always null. Cost statistics are returned by the comparison endpoint. - `control_avg_duration` number, nullable: Average duration of completed control runs, in milliseconds. - `variant_avg_duration` number, nullable: Average duration of completed variant runs, in milliseconds. - `control_avg_score` number, nullable: Average normalized judge score (0 to 1) of control runs, from the metric or quality judge when set. null without evaluations. - `variant_avg_score` number, nullable: Average normalized judge score (0 to 1) of variant runs, from the metric or quality judge when set. null without evaluations. - `control` PublicABTestArmStatsResponse, required - `count` integer, required: Completed runs in this arm. - `token_cost_statistics` object, required: Cost per run keyed by currency code, over finished runs whose calls could all be priced. Judge costs are excluded. - `token_cost_coverage` PublicTokenCostCoverageResponse, required: How many model calls could be priced. - `complete` boolean, required: True when every model call could be priced. - `call_count` integer, required: Model calls covered by the report. - `priced_call_count` integer, required: Calls included in the cost amounts. - `unpriced_call_count` integer, required: Calls without a known price or without reported token usage. They are left out of the amounts. - `unpriced_models` string[], required: Models of the unpriced calls. - `avg_duration` number, nullable, required: Mean duration of completed runs in milliseconds. Null when there are none. - `avg_score` number, nullable, required: Average judge score normalized to 0 to 1 (score divided by maximum score), using the latest evaluation per run and judge. Null when no completed run was scored. - `success_rate` number, nullable, required: Percentage of finished runs that completed, from 0 to 100. Null when no run has finished. - `p50_duration` number, nullable, required: Median duration of completed runs in milliseconds. Null when there are none. - `p95_duration` number, nullable, required: 95th percentile duration of completed runs in milliseconds. Null when there are none. - `judge_evaluated_count` integer, required: Completed runs with at least one judge score. - `judge_coverage` number, nullable, required: Percentage of completed runs with a judge score, from 0 to 100. Null when there are no completed runs. - `variant` PublicABTestArmStatsResponse, required - `count` integer, required: Completed runs in this arm. - `token_cost_statistics` object, required: Cost per run keyed by currency code, over finished runs whose calls could all be priced. Judge costs are excluded. - `token_cost_coverage` PublicTokenCostCoverageResponse, required: How many model calls could be priced. - `complete` boolean, required: True when every model call could be priced. - `call_count` integer, required: Model calls covered by the report. - `priced_call_count` integer, required: Calls included in the cost amounts. - `unpriced_call_count` integer, required: Calls without a known price or without reported token usage. They are left out of the amounts. - `unpriced_models` string[], required: Models of the unpriced calls. - `avg_duration` number, nullable, required: Mean duration of completed runs in milliseconds. Null when there are none. - `avg_score` number, nullable, required: Average judge score normalized to 0 to 1 (score divided by maximum score), using the latest evaluation per run and judge. Null when no completed run was scored. - `success_rate` number, nullable, required: Percentage of finished runs that completed, from 0 to 100. Null when no run has finished. - `p50_duration` number, nullable, required: Median duration of completed runs in milliseconds. Null when there are none. - `p95_duration` number, nullable, required: 95th percentile duration of completed runs in milliseconds. Null when there are none. - `judge_evaluated_count` integer, required: Completed runs with at least one judge score. - `judge_coverage` number, nullable, required: Percentage of completed runs with a judge score, from 0 to 100. Null when there are no completed runs. - `has_enough_data` boolean, required: True when both arms have at least `min_sample_size` completed runs. - `min_sample_size` integer, required: Minimum completed runs per arm set on the test. - `analysis` PublicABTestAnalysisResponse, required: Sequential confidence analysis of the test's primary metric. - `recommendation` 'collecting' | 'continue' | 'variant_recommended' | 'control_recommended' | 'inconclusive' | 'invalid_srm' | 'safety_stopped', required: Test verdict: `collecting` before the first planned look, `continue` after a look without a decision, `variant_recommended` or `control_recommended` when a boundary was crossed, `inconclusive`, `invalid_srm` after a sample-ratio mismatch, or `safety_stopped` when the test was stopped automatically. - `reason` string, nullable, required: Explanation for the recommendation. Null when there is none. - `validity` PublicABTestValidityResponse, required: Sample ratio mismatch check: whether runs were split between the arms as planned. - `status` 'invalid' | 'valid' | 'pending', required: `valid` when the latest planned look passed the check, `invalid` when the split differs from the planned allocation more than chance allows and the comparison cannot be trusted, `pending` before the first look. - `srm_p_value` number, nullable, required: P-value of a chi-square test comparing the observed split with `expected_variant_allocation` at the latest look. Values below `threshold` flag a mismatch; null before the first look. - `threshold` number, required: P-value below which a sample ratio mismatch is reported. - `control_assignments` integer, required: Control assignments counted in the check, or assigned so far before the first look. - `variant_assignments` integer, required: Variant assignments counted in the check, or assigned so far before the first look. - `expected_variant_allocation` number, required: Planned share of assignments sent to the variant, from 0 to 1. - `primary_metric` PublicABTestPrimaryMetricResponse, required: Primary metric estimate for both arms and the difference between them. - `metric` 'success_rate' | 'judge_score', required: `success_rate`: share of runs that completed. `judge_score`: judge score normalized to 0 to 1, with runs that did not complete scored 0. - `control_value` number, nullable, required: Metric value for the control arm. Null until both arms have data. - `variant_value` number, nullable, required: Metric value for the variant arm. Null until both arms have data. - `effect_size` number, nullable, required: Absolute difference, variant minus control. - `relative_effect` number, nullable, required: Difference relative to the control value, where 0.1 means 10% higher. Null when the control value is 0. - `ci_lower` number, nullable, required: Lower bound of the confidence interval for `effect_size`. Null until the first planned look. - `ci_upper` number, nullable, required: Upper bound of the confidence interval for `effect_size`. Null until the first planned look. - `confidence_level` number, required: Confidence level of the interval, 1 minus the test's alpha (for example 0.95). - `interval_type` 'descriptive' | 'sequential_adjusted', required: `sequential_adjusted`: values come from the latest planned look, with the interval widened to stay valid across repeated looks. `descriptive`: point estimates from the data so far, without an interval. - `control_observed` integer, required: Control assignments with a recorded outcome so far, up to `control_target`. - `variant_observed` integer, required: Variant assignments with a recorded outcome so far, up to `variant_target`. - `control_analyzed` integer, required: Control outcomes used at the latest planned look. 0 before the first look. - `variant_analyzed` integer, required: Variant outcomes used at the latest planned look. 0 before the first look. - `control_target` integer, required: Planned number of control outcomes. - `variant_target` integer, required: Planned number of variant outcomes. - `information_fraction` number, required: Share of the planned sample collected so far, from 0 to 1: the lower of the two arms' observed-to-target ratios. - `latest_look` integer, nullable, required: Number of the latest planned look that has run. Null before the first look. - `next_look_fraction` number, nullable, required: Information fraction at which the next planned look runs. Null when all looks are done. - `missing_outcomes` integer, required: Assignments without a usable outcome yet, for example because the run is still active or has no judge score. - `judge_evaluation_errors` integer, required: Assignments whose primary-metric judge evaluation failed. They are left out of the analysis. - `cost_comparison` PublicABTestCostComparisonResponse, required - `by_currency` object, required: Comparison for each currency code present in both arms. Empty when either arm has model calls that could not be priced. - `overall_winner` 'control' | 'variant' | 'tie', nullable, required: Winner when both arms were priced in one and the same currency. Null otherwise. - `selected_judge_id` string, uuid, nullable: Judge whose scores fill `avg_score`: the primary-metric judge when the primary metric is `judge_score`, otherwise the quality judge. Present only when the caller can view judges. - `selected_judge_name` string, nullable: Name of the judge in `selected_judge_id`. Present only when the caller can view judges. - `quality_judge_name` string, nullable: Name of the test's quality judge. Present only when the caller can view judges. ## Other responses - `401`: Missing or invalid API key. - `403`: The API key lacks the required permission or access to the environment. - `404`: The resource does not exist in this project. - `422`: Validation Error - `429`: Rate limit exceeded. --- [API](https://skmtc.dev/connic/apis/connic-api.md) · [All operations](https://skmtc.dev/connic/apis/connic-api/llms.txt) · [OpenAPI document](https://skmtc.dev/connic/apis/connic-api/revisions/482932da5fbe?raw)