---
title: "Create judge"
method: POST
path: "/v1/projects/{project_id}/judges"
tags: ["judges"]
---

# Create judge

`POST /v1/projects/{project_id}/judges`

## Request

- Base URL: `https://api.connic.co`
- URL: `https://api.connic.co/v1/projects/{project_id}/judges`
- Auth: HTTP bearer

Creates a judge that scores runs of one agent in the given environment against a rubric, using an LLM. With `trigger_mode=automatic`, it evaluates completed runs that match `filter_expression`, sampled at `sample_rate`; with `manual`, evaluations start only through the trigger endpoints. The judge only evaluates runs of the `agent_name` in the body.

Required permission: `judges.create`.

## Path parameters

- `project_id` string, uuid, required

## Query parameters

- `environment_id` string, uuid, required: Environment the judge belongs to

## Request body

- JudgeCreate
  - `name` string, required: Display name, at most 200 characters.
  - `agent_name` string, required: Agent whose runs the judge evaluates.
  - `model` string, required: Model that scores runs, for example a `connic/*` model. At most 512 characters.
  - `system_prompt` string: Extra instructions for the scoring model. At most 8,000 characters.
  - `include_agent_system_prompt` boolean: Include the evaluated agent's system prompt in the scoring context.
  - `criteria` JudgeCriterion[], required: Scoring rubric with 1 to 20 criteria. The combined `max_score` is at most 10,000 and the combined name and description length at most 20,000 characters.
    - `name` string, required: Criterion name, unique within the judge (case-insensitive). At most 200 characters.
    - `description` string, required: What the scoring model evaluates for this criterion. At most 2,000 characters.
    - `max_score` number, required: Highest score for this criterion, greater than 0 and at most 1,000.
  - `filter_expression` string: Expression over the run's `input`, `output` and `context` that must be true for automatic evaluation. Empty matches every run.
  - `trigger_mode` 'automatic' | 'manual': `automatic` evaluates runs when they complete; `manual` evaluates only when triggered.
  - `sample_rate` number: Share of matching runs evaluated automatically, from 0 to 1.
  - `notification_threshold` number, nullable: Sends a `judge_score_low` notification when the average score, as a share of the maximum, drops below this value (0 to 1). null disables it.
  - `notification_window` integer, nullable: Number of most recent evaluations averaged for `notification_threshold` (1 to 10,000). null averages all.
  - `enabled` boolean: When false, runs are not evaluated automatically; on-demand triggers still work.

## Response `201`

Successful Response

- JudgeResponse: An LLM judge that scores one agent's runs against a rubric, with evaluation totals.
  - `id` string, uuid, required
  - `project_id` string, uuid, required: Project the judge belongs to.
  - `environment_id` string, uuid, required: Environment whose runs the judge evaluates.
  - `name` string, required
  - `agent_name` string, required: Agent whose runs the judge evaluates.
  - `model` string, required: Model that scores runs.
  - `system_prompt` string, required: Extra instructions for the scoring model.
  - `include_agent_system_prompt` boolean, required: Whether the evaluated agent's system prompt is included in the scoring context.
  - `criteria` JudgeCriterion[], required: Scoring rubric.
    - `name` string, required: Criterion name, unique within the judge (case-insensitive). At most 200 characters.
    - `description` string, required: What the scoring model evaluates for this criterion. At most 2,000 characters.
    - `max_score` number, required: Highest score for this criterion, greater than 0 and at most 1,000.
  - `filter_expression` string, required: Expression a run must match to be evaluated automatically. Empty matches every run.
  - `trigger_mode` string, required: `automatic` evaluates runs when they complete; `manual` evaluates only when triggered.
  - `sample_rate` number, required: Share of matching runs evaluated automatically, from 0 to 1.
  - `notification_threshold` number, nullable: Average score share (0 to 1) below which a `judge_score_low` notification is sent. null when disabled.
  - `notification_window` integer, nullable: Number of most recent evaluations averaged for `notification_threshold`. null averages all.
  - `enabled` boolean, required: When false, runs are not evaluated automatically.
  - `created_at` string, date-time, required
  - `updated_at` string, date-time, required
  - `avg_score` number, nullable: Average overall score of completed evaluations. null before the first one.
  - `avg_max_score` number, nullable: Average maximum possible score of completed evaluations. null before the first one.
  - `total_evaluated` integer: Number of completed evaluations.

## Other responses

- `401`: Missing or invalid API key.
- `403`: The API key lacks the required permission or access to the environment.
- `422`: Validation Error
- `429`: Rate limit exceeded.

---

[API](https://skmtc.dev/connic/apis/connic-api.md) · [All operations](https://skmtc.dev/connic/apis/connic-api/llms.txt) · [OpenAPI document](https://skmtc.dev/connic/apis/connic-api/revisions/482932da5fbe?raw)