Judge Complementarity

Does the human + judge TEAM beat either of them alone?

The sibling /agreement and /disagreements endpoints measure substitution: how closely the judge reproduces the human. Both peak when the judge is a perfect replica — which is worth nothing to a team, because a replica never rescues a pair the human got wrong.

This endpoint measures the other axis. It scores a single held-out rater and the judge against the same leave-one-out consensus of the remaining raters, splits the outcomes into a 2x2 (the off-diagonals are where complementarity lives), and sweeps a deferral threshold — the machine answers when its confidence clears τ, a human answers otherwise — to find what the pair achieves together.

With no judge_id the ranking's most prolific judge is used and named in the response. A hardcoded default would report "insufficient" on a ranking judged by anything else, which reads as thin data rather than as a wrong argument.

Reads persisted votes only: no judge calls, so nothing is metered.