Benchmark

Queries a benchmark's image matchups — a page of compared image pairs with how the responses split between them.

An item is one image pair as one leaderboard saw it, so a pair compared on several leaderboards comes back once per leaderboard and the boards' verdicts on it can be read side by side. Ordering (sigma by default) ranks the matchups themselves — sigma being how far apart the two images' records over all of their comparisons on the prompt sit, in standard errors of their logit gap — where the prompt-ratings query ranks prompts by their most decisive matchup. Filters use that query's field set, applied to the votes a matchup is tallied from, plus WinnerParticipantId and LooserParticipantId, which select matchups by which side won the whole matchup; those two are only accepted as top-level and leaves, and a tied matchup matches neither. Matchups of leaderboards excluded from the benchmark overall are never returned. sides holds exactly two images, the one that took more responses first, and their responses sum to the matchup's. Each side carries its theta — the pooled per-image record sigma is derived from. extremeness and the winning side are always read off the raw counts, and sigma/theta are always weighted by annotator reliability; useWeightedScoring changes only each side's wins.

get/benchmark/{benchmarkId}/matchups/query

Path parameters

benchmarkIdstring required

The id of the benchmark.

Query parameters

useWeightedScoringboolean

Whether to report each side's wins weighted by annotator reliability.

groupBy'Image' | 'Participant'

What each rated entry of a prompt represents.

What a compared image stands for. Per image, every matchup is returned. Per participant, a matchup between two images of one participant is a self-matchup and leaves the page.

pageinteger

The 1-based page index.

page_sizeinteger

The number of items per page.

sortstring[]

Sort fields. Prefix with - for descending order (e.g. -created_at).

eqstring
neqstring
gtstring
gtestring
ltstring
ltestring
containsstring
starts_withstring
ends_withstring
instring[]
not_containsstring

Filter by country.

eqstring
neqstring
gtstring
gtestring
ltstring
ltestring
containsstring
starts_withstring
ends_withstring
instring[]
not_containsstring

Filter by language.

eqstring
neqstring
gtstring
gtestring
ltstring
ltestring
containsstring
starts_withstring
ends_withstring
instring[]
not_containsstring

Filter by gender.

eqstring
neqstring
gtstring
gtestring
ltstring
ltestring
containsstring
starts_withstring
ends_withstring
instring[]
not_containsstring

Filter by age_bucket.

eqstring
neqstring
gtstring
gtestring
ltstring
ltestring
containsstring
starts_withstring
ends_withstring
instring[]
not_containsstring

Filter by occupation.

eqstring
neqstring
gtstring
gtestring
ltstring
ltestring
containsstring
starts_withstring
ends_withstring
instring[]
not_containsstring

Filter by tags.

eqstring
neqstring
gtstring
gtestring
ltstring
ltestring
containsstring
starts_withstring
ends_withstring
instring[]
not_containsstring

Filter by participant_id.

eqstring
neqstring
gtstring
gtestring
ltstring
ltestring
containsstring
starts_withstring
ends_withstring
instring[]
not_containsstring

Filter by leaderboard_id.

eqstring
neqstring
gtstring
gtestring
ltstring
ltestring
containsstring
starts_withstring
ends_withstring
instring[]
not_containsstring

Filter by run_id.

eqstring
neqstring
gtstring
gtestring
ltstring
ltestring
containsstring
starts_withstring
ends_withstring
instring[]
not_containsstring

Filter by prompt_identifier.

eqstring
neqstring
gtstring
gtestring
ltstring
ltestring
containsstring
starts_withstring
ends_withstring
instring[]
not_containsstring

Filter by voted_at.

eqstring
neqstring
gtstring
gtestring
ltstring
ltestring
containsstring
starts_withstring
ends_withstring
instring[]
not_containsstring

Filter by winner_participant_id.

eqstring
neqstring
gtstring
gtestring
ltstring
ltestring
containsstring
starts_withstring
ends_withstring
instring[]
not_containsstring

Filter by looser_participant_id.

logic'and' | 'or'

How to combine the field filters: "and" (default) requires every filter to match, "or" requires any of them to match.

Response

OK

totalinteger required

The number of matchups the filter leaves — the paging total.

pageinteger required

The 1-based page index.

pageSizeinteger required

The page size in matchups.

Changes