LLM Judges

Judge Vote

Have an LLM judge vote on a single matchup.

Owner-only: only the ranking's owner can invoke paid LLM calls on it.

post/api/v1/judges/vote

Request

  • Base URL: https://app.aimrank.io (the document declares no server; this is the origin it was published from)
  • URL: https://app.aimrank.io/api/v1/judges/vote
  • Auth: none declared

Request body

ranking_idstring required
entity_a_idstring required
entity_b_idstring required
judge_idstring
criterionstring nullable

What the judge should optimize for. Defaults to the ranking's name/description.

randomize_positionboolean

Run A/B and B/A and reconcile — mitigates position bias. 2× the cost.

submit_as_voteboolean

If true, record the verdict as a vote. If false, return the verdict without persisting.

max_tokensinteger

Output token cap. Default 400 fits the JSON verdict; bump to 800–2000 for verbose rationale used in interpretability training.

thinkingboolean

Enable Anthropic extended thinking on Opus 4.7. Adds ~5–10pp accuracy on adversarial subsets at 2–5× cost. Silently no-op on Sonnet/Haiku because they don't support the field.

apply_constitutionboolean

Prepend the ranking's effective constitution (Phase 9.2 alignment scaffold) to the judge prompt — so the judge applies the same principles humans see in the UI. Set false to bypass.

Response

Successful Response

unknown required

Changes

No recorded changes to this operation across all 1 revision of this API.