Get Ranking Agreement
Inter-annotator agreement metrics for a ranking.
Returns Cohen's κ (if exactly 2 voters), Fleiss' κ (N voters), Krippendorff's α (the preferred nominal-data IAA metric), and Scott's π, plus a Landis-Koch interpretation bucket.
Values of None mean the dataset has insufficient coverage (e.g., no matchup has received ≥ 2 votes) — which is honest rather than misleading. Used by AI-lab buyers who need to know whether our preference data is reliable before training a reward model on it.
If the ranking has min_raters_per_pair > 1, pairs below that threshold are excluded from the IAA computation (and the response reports both pre- and post-filter coverage so buyers can see the cost of the cut).
Request
- Base URL: https://app.aimrank.io (the document declares no server; this is the origin it was published from)
- URL: https://app.aimrank.io/api/v1/analytics/rankings/{ranking_id}/agreement
- Auth: none declared
Path parameters
Query parameters
Override the ranking's min_raters_per_pair when computing IAA. Defaults to the ranking's configured value.
Response
Successful Response
Changes
No recorded changes to this operation across all 1 revision of this API.