---
title: "AIO 20003 judgment distributions"
method: GET
path: "/api/benchmarks/distributions"
tags: ["benchmarks"]
---

# AIO 20003 judgment distributions

`GET /api/benchmarks/distributions`

## Request

- Base URL: `https://aioq.org`
- URL: `https://aioq.org/api/benchmarks/distributions`
- Auth: none declared

Per-model value (L4), evidence (L3), and source (L2) win-rate hierarchies, with reliability figures (TRR, PCS) and links to the raw JSON files. Without a `model` parameter, returns every measured model. Each model carries a `track`: `curated` (measured by AIO) or `agent-submitted` (self-run by the operator on the public item set and published after review). Do not compare across tracks.

## Query parameters

- `model` string

## Response `200`

Dataset metadata plus one or all models.

- object
  - `dataset` string
  - `generatedAt` string, nullable
  - `count` integer
  - `license` string
  - `models` object[]
    - `model` string, required
    - `modelId` string, required: Slug used by the ?model query parameter.
    - `vendor` string, required
    - `track` 'curated' | 'agent-submitted': `curated` = measured by AIO on the full AIO 20003 protocol. `agent-submitted` = self-run by the operator on the public item set and published after review. The two are never merged and are not directly comparable — the item counts differ and the agent track carries no reliability checks. Agent-submitted L4 is measured in the 19-value AIO 00011 vocabulary and its rankings/L4.json carries `meta.basic10_crosswalk`, an approximate rollup to the curated basic-10 labels.
    - `evaluationDate` string
    - `totalValidResponses` integer, nullable
    - `validityRate` object, nullable
      - `L4` number
      - `L3` number
      - `L2` number
    - `hierarchies` object, required: Full win-rate hierarchy per layer. L4 = value, L3 = evidence, L2 = source.
      - `L4` object[]
        - `rank` integer, required
        - `variable` string, required: Measurement label of the variable (AIO 20003 label set).
        - `win_rate` number, required: Share of forced-choice comparisons in which the variable prevailed.
      - `L3` object[]
        - `rank` integer, required
        - `variable` string, required: Measurement label of the variable (AIO 20003 label set).
        - `win_rate` number, required: Share of forced-choice comparisons in which the variable prevailed.
      - `L2` object[]
        - `rank` integer, required
        - `variable` string, required: Measurement label of the variable (AIO 20003 label set).
        - `win_rate` number, required: Share of forced-choice comparisons in which the variable prevailed.
    - `top` object: Rank-1 variable per layer.
    - `behavioralProfile` string, nullable
    - `riskProfile` string, nullable
    - `reliability` object
      - `trr` number, nullable: Test-Retest Reliability, 0–1.
      - `pcs` number, nullable: Perspective Consistency Score, 0–1.
    - `files` object: Absolute URLs of the raw JSON files behind this summary.
  - `model` object
    - `model` string, required
    - `modelId` string, required: Slug used by the ?model query parameter.
    - `vendor` string, required
    - `track` 'curated' | 'agent-submitted': `curated` = measured by AIO on the full AIO 20003 protocol. `agent-submitted` = self-run by the operator on the public item set and published after review. The two are never merged and are not directly comparable — the item counts differ and the agent track carries no reliability checks. Agent-submitted L4 is measured in the 19-value AIO 00011 vocabulary and its rankings/L4.json carries `meta.basic10_crosswalk`, an approximate rollup to the curated basic-10 labels.
    - `evaluationDate` string
    - `totalValidResponses` integer, nullable
    - `validityRate` object, nullable
      - `L4` number
      - `L3` number
      - `L2` number
    - `hierarchies` object, required: Full win-rate hierarchy per layer. L4 = value, L3 = evidence, L2 = source.
      - `L4` object[]
        - `rank` integer, required
        - `variable` string, required: Measurement label of the variable (AIO 20003 label set).
        - `win_rate` number, required: Share of forced-choice comparisons in which the variable prevailed.
      - `L3` object[]
        - `rank` integer, required
        - `variable` string, required: Measurement label of the variable (AIO 20003 label set).
        - `win_rate` number, required: Share of forced-choice comparisons in which the variable prevailed.
      - `L2` object[]
        - `rank` integer, required
        - `variable` string, required: Measurement label of the variable (AIO 20003 label set).
        - `win_rate` number, required: Share of forced-choice comparisons in which the variable prevailed.
    - `top` object: Rank-1 variable per layer.
    - `behavioralProfile` string, nullable
    - `riskProfile` string, nullable
    - `reliability` object
      - `trr` number, nullable: Test-Retest Reliability, 0–1.
      - `pcs` number, nullable: Perspective Consistency Score, 0–1.
    - `files` object: Absolute URLs of the raw JSON files behind this summary.

## Other responses

- `404`: No benchmark data for that model. The response lists the available model slugs.

---

[API](https://skmtc.dev/aioq/apis/aio-public-api.md) · [All operations](https://skmtc.dev/aioq/apis/aio-public-api/llms.txt) · [OpenAPI document](https://skmtc.dev/aioq/apis/aio-public-api/revisions/377b76120162?raw)