---
title: "Create a scorer"
method: POST
path: "/factory/{uid}/scorers"
tags: ["Factories", "Scorers"]
---

# Create a scorer

`POST /factory/{uid}/scorers`

## Request

- Base URL: `https://app.warp.dev/api/v1`
- URL: `https://app.warp.dev/api/v1/factory/{uid}/scorers`
- Auth: HTTP bearer

Create an active run scorer for a factory with either selected-agent or
all-agent scope. Creating a scorer does not start scoring. Pass
self_improvement_enabled to also turn on self-improvement for the new scorer in the same
request; the scorer and its self-improvement config are created atomically.

## Request body

- CreateScorerRequest: Scorer definition. Text fields are trimmed of surrounding whitespace before validation; required text fields must not be blank after trimming, and no text field may exceed 1 MiB (1048576 bytes).
  - `factory_uid` string: Optional compatibility scope; must match the factory_uid in the path.
  - `name` string, required: Display name for the scorer
  - `description` string, nullable: Optional description of the scorer
  - `scoring_prompt` string, required: Instructions used to score matching runs
  - `allowed_classifications` ScorerClassification[], required: Values the scorer may return; classification values must be unique
    - `value` string, required: Classification value returned by the scorer
    - `description` string, nullable: Optional free-text meaning of this classification, surfaced to the judge alongside the value. Omit or leave blank for a score-only label.
    - `score` number, double, required: Score this classification carries, from 0 to 1. A run passes when its scored label's score is greater than or equal to the scorer's threshold. A score of exactly 0 is valid, so this is not enforced with a "required" binding (which would reject the zero value); handlers validate its range explicitly instead.
  - `threshold` number, double, required: Score a run's classified label must meet or exceed to pass, from 0 to 1.
  - `scope_mode` 'all_agents' | 'selected_agents' | 'selected_stages', required: Whether the scorer applies to every factory agent, selected agents, or selected blueprint stages
  - `agent_uids` string[]: Legacy shorthand for selected agents with include_descendants=false. Required and non-empty for selected_agents when agents is omitted; must be empty for all_agents and cannot be combined with agents.
  - `agents` ScorerAgentInput[]: Selected agents and their evidence policy. Required and non-empty for selected_agents when agent_uids is omitted; must be empty for all_agents or selected_stages and cannot be combined with agent_uids.
    - `uid` string, required: Unique identifier of the selected agent
    - `include_descendants` boolean: Whether scoring this agent's run includes its descendant subtree
  - `stages` string[]: Blueprint stage names. Required for selected_stages and mutually exclusive with agent_uids and agents. Available only for blueprint factories.
  - `sampling_rate` number, double, nullable: Percentage of the scorer's eligible runs to score, from 0 to 100; omit to score every eligible run, and 0 stops automatic scoring (manual dispatch still works). A value with more than two decimal places is rounded to two rather than rejected, and the rounded value is what is stored. Sampling applies to periodic scoring only, and runs are chosen deterministically per (scorer, run), so lowering the rate reduces how many runs are scored rather than how often.
  - `model_id` string, required: LLM model dispatched judge runs use to evaluate this scorer's rubric.
  - `self_improvement_enabled` boolean: Optionally enable self-improvement for the newly created scorer in the same transactional request. Defaults to false; the response includes the resulting state.
  - `agent_config` ScorerAgentConfig: Scorer-specific execution overrides for its hidden judge agent. An omitted or null field inherits the corresponding Factory agent default. A non-empty value replaces that default. Empty secrets and mcp_servers collections explicitly clear the optional Factory defaults.
    - `default_runner_uid` string, nullable: Runner used by scorer judge runs. Omit or set null to inherit the Factory agent default.
    - `secrets` SecretRef[], nullable: Managed secrets attached to scorer judge runs. Omit or set null to inherit the Factory agent defaults; an empty array attaches none.
      - `name` string, required: Name of the managed secret.
    - `mcp_servers` object, nullable: MCP servers attached to scorer judge runs. Omit or set null to inherit the Factory agent defaults; an empty object attaches none.

## Response `201`

Scorer created successfully

- ScorerResponse
  - `scorer_uid` string, uuid, required: Stable scorer UUID; new scorers use UUIDv4. Nil UUIDs are invalid.
  - `self_improvement_enabled` boolean, required: Whether self-improvement is enabled for this scorer
  - `id` integer, required: Scorer identifier
  - `factory_uid` string, required: UID of the factory that owns the scorer
  - `name` string, required: Display name for the scorer
  - `description` string, required: Description of the scorer
  - `scoring_prompt` string, required: Instructions used to score matching runs
  - `allowed_classifications` ScorerClassification[], required
    - `value` string, required: Classification value returned by the scorer
    - `description` string, nullable: Optional free-text meaning of this classification, surfaced to the judge alongside the value. Omit or leave blank for a score-only label.
    - `score` number, double, required: Score this classification carries, from 0 to 1. A run passes when its scored label's score is greater than or equal to the scorer's threshold. A score of exactly 0 is valid, so this is not enforced with a "required" binding (which would reject the zero value); handlers validate its range explicitly instead.
  - `threshold` number, double, required: Score a run's classified label must meet or exceed to pass, from 0 to 1
  - `scorer_kind` 'user' | 'benchmark_task_correctness', required: Whether the scorer is user-defined or a platform-owned managed scorer. Read-only on responses; create input does not accept scorer_kind.
  - `scope_mode` 'all_agents' | 'selected_agents' | 'selected_stages', required: Whether the scorer applies to every factory agent, selected agents, or selected blueprint stages
  - `stages` string[], required: Blueprint stage names the scorer applies to; empty unless scope_mode is selected_stages
  - `agent_uids` string[], required: Agents attached to the scorer; empty unless scope_mode is selected_agents
  - `agents` ScorerAgentInput[], required: Selected agents and whether each evaluation includes its run descendants; empty unless scope_mode is selected_agents
    - `uid` string, required: Unique identifier of the selected agent
    - `include_descendants` boolean: Whether scoring this agent's run includes its descendant subtree
  - `sampling_rate` number, double, required: Percentage of the scorer's eligible runs that periodic scoring scores, to two decimal places. 0 stops automatic scoring.
  - `model_id` string, required: LLM model dispatched judge runs use to evaluate this scorer's rubric.
  - `agent_config` ScorerAgentConfig, required: Scorer-specific execution overrides for its hidden judge agent. An omitted or null field inherits the corresponding Factory agent default. A non-empty value replaces that default. Empty secrets and mcp_servers collections explicitly clear the optional Factory defaults.
    - `default_runner_uid` string, nullable: Runner used by scorer judge runs. Omit or set null to inherit the Factory agent default.
    - `secrets` SecretRef[], nullable: Managed secrets attached to scorer judge runs. Omit or set null to inherit the Factory agent defaults; an empty array attaches none.
      - `name` string, required: Name of the managed secret.
    - `mcp_servers` object, nullable: MCP servers attached to scorer judge runs. Omit or set null to inherit the Factory agent defaults; an empty object attaches none.
  - `created_at` string, date-time, required: When the scorer was created
  - `updated_at` string, date-time, required: When the scorer was updated

## Other responses

- `400`: Invalid scorer definition
- `401`: Authentication required
- `403`: Caller cannot edit the factory
- `404`: Factory not found
- `500`: Internal server error

## Changes

- **2026-11-04** `4b5d145669be`: 1 warning, 3 info
  - added the new `selected_stages` enum value to the `scope_mode` response property for the response status `201`
  - added the new optional request property `stages`
  - added the new `selected_stages` enum value to the request property `scope_mode`
  - added the required property `stages` to the response with the `201` status
- **2026-10-21** `7ee3ba55debc`: 2 info
  - api tag `factory-scorers` added
  - api tag `factory` removed
- **2026-10-19** `2c45dcc0a13e`: 6 breaking
  - the request property `agent_config/default_runner_uid` became not nullable
  - the request property `agent_config/mcp_servers` became not nullable
  - the request property `agent_config/secrets` became not nullable
  - the request property `allowed_classifications/items/description` became not nullable
  - …2 more
- **2026-10-15** `f4630aae17ba`: 1 info
  - endpoint added

[Operation history](https://skmtc.dev/warp/apis/oz-agent-api/changes/factory/:uid/scorers/post.md)

---

[API](https://skmtc.dev/warp/apis/oz-agent-api.md) - [All operations](https://skmtc.dev/warp/apis/oz-agent-api/llms.txt) - [OpenAPI document](https://skmtc.dev/warp/apis/oz-agent-api/revisions/4b5d145669be?raw)