Judge Disagreements

LLM-vs-human alignment for one judge in a single call.

Computes the human majority winner per matchup, the judge's pick per matchup, Cohen κ over the overlap, a Wilson 95% CI on the agreement rate, and the top matchups where the judge disagrees with the human majority.

Cohen κ on the a/b-relative encoding (did the rater pick the lower-sorted entity). Returns null when a rater's marginal is degenerate (e.g. everyone always prefers the stronger entity) — in that case agreement_pct + the Wilson 95% CI are the primary alignment signal.