judges

Create judge

Creates a judge that scores runs of one agent in the given environment against a rubric, using an LLM. With trigger_mode=automatic, it evaluates completed runs that match filter_expression, sampled at sample_rate; with manual, evaluations start only through the trigger endpoints. The judge only evaluates runs of the agent_name in the body.

Required permission: judges.create.

post/v1/projects/{project_id}/judges

Request

  • Base URL: https://api.connic.co
  • URL: https://api.connic.co/v1/projects/{project_id}/judges
  • Auth: HTTP bearer

Path parameters

project_idstring uuid required

ID of the project the API key belongs to.

Query parameters

environment_idstring uuid required

Environment the judge belongs to

Request body

namestring required

Display name, at most 200 characters.

agent_namestring required

Agent whose runs the judge evaluates.

modelstring required

Model that scores runs, for example a connic/* model. At most 512 characters.

system_promptstring

Extra instructions for the scoring model. At most 8,000 characters.

include_agent_system_promptboolean

Include the evaluated agent's system prompt in the scoring context.

filter_expressionstring

Expression over the run's input, output and context that must be true for automatic evaluation. Empty matches every run.

trigger_mode'automatic' | 'manual'

automatic evaluates runs when they complete; manual evaluates only when triggered.

sample_ratenumber

Share of matching runs evaluated automatically, from 0 to 1.

notification_thresholdnumber nullable

Sends a judge_score_low notification when the average score, as a share of the maximum, drops below this value (0 to 1). null disables it.

notification_windowinteger nullable

Number of most recent evaluations averaged for notification_threshold (1 to 10,000). null averages all.

enabledboolean

When false, runs are not evaluated automatically; on-demand triggers still work.

Response

Successful Response

idstring uuid required
project_idstring uuid required

Project the judge belongs to.

environment_idstring uuid required

Environment whose runs the judge evaluates.

namestring required
agent_namestring required

Agent whose runs the judge evaluates.

modelstring required

Model that scores runs.

system_promptstring required

Extra instructions for the scoring model.

include_agent_system_promptboolean required

Whether the evaluated agent's system prompt is included in the scoring context.

filter_expressionstring required

Expression a run must match to be evaluated automatically. Empty matches every run.

trigger_modestring required

automatic evaluates runs when they complete; manual evaluates only when triggered.

sample_ratenumber required

Share of matching runs evaluated automatically, from 0 to 1.

notification_thresholdnumber nullable

Average score share (0 to 1) below which a judge_score_low notification is sent. null when disabled.

notification_windowinteger nullable

Number of most recent evaluations averaged for notification_threshold. null averages all.

enabledboolean required

When false, runs are not evaluated automatically.

created_atstring date-time required
updated_atstring date-time required
avg_scorenumber nullable

Average overall score of completed evaluations. null before the first one.

avg_max_scorenumber nullable

Average maximum possible score of completed evaluations. null before the first one.

total_evaluatedinteger

Number of completed evaluations.

Changes

No recorded changes to this operation across all 1 revision of this API.