WAwarp
FactoriesScorers

Create a scorer

Changed on

Create an active run scorer for a factory with either selected-agent or all-agent scope. Creating a scorer does not start scoring. Pass self_improvement_enabled to also turn on self-improvement for the new scorer in the same request; the scorer and its self-improvement config are created atomically.

post/factory/{uid}/scorers

Request

  • Base URL
    https://app.warp.dev/api/v1
  • Auth
    HTTP bearer

Request body

factory_uidstring

Optional compatibility scope; must match the factory_uid in the path.

namestring required

Display name for the scorer

descriptionstring nullable

Optional description of the scorer

scoring_promptstring required

Instructions used to score matching runs

thresholdnumber double required

Score a run's classified label must meet or exceed to pass, from 0 to 1.

scope_mode'all_agents' | 'selected_agents' | 'selected_stages' required

Whether the scorer applies to every factory agent, selected agents, or selected blueprint stages

agent_uidsstring[]

Legacy shorthand for selected agents with include_descendants=false. Required and non-empty for selected_agents when agents is omitted; must be empty for all_agents and cannot be combined with agents.

stagesstring[]

Blueprint stage names. Required for selected_stages and mutually exclusive with agent_uids and agents. Available only for blueprint factories.

sampling_ratenumber double nullable

Percentage of the scorer's eligible runs to score, from 0 to 100; omit to score every eligible run, and 0 stops automatic scoring (manual dispatch still works). A value with more than two decimal places is rounded to two rather than rejected, and the rounded value is what is stored. Sampling applies to periodic scoring only, and runs are chosen deterministically per (scorer, run), so lowering the rate reduces how many runs are scored rather than how often.

model_idstring required

LLM model dispatched judge runs use to evaluate this scorer's rubric.

self_improvement_enabledboolean

Optionally enable self-improvement for the newly created scorer in the same transactional request. Defaults to false; the response includes the resulting state.

Response

Scorer created successfully

scorer_uidstring uuid required

Stable scorer UUID; new scorers use UUIDv4. Nil UUIDs are invalid.

self_improvement_enabledboolean required

Whether self-improvement is enabled for this scorer

idinteger required

Scorer identifier

factory_uidstring required

UID of the factory that owns the scorer

namestring required

Display name for the scorer

descriptionstring required

Description of the scorer

scoring_promptstring required

Instructions used to score matching runs

thresholdnumber double required

Score a run's classified label must meet or exceed to pass, from 0 to 1

scorer_kind'user' | 'benchmark_task_correctness' required

Whether the scorer is user-defined or a platform-owned managed scorer. Read-only on responses; create input does not accept scorer_kind.

scope_mode'all_agents' | 'selected_agents' | 'selected_stages' required

Whether the scorer applies to every factory agent, selected agents, or selected blueprint stages

stagesstring[] required

Blueprint stage names the scorer applies to; empty unless scope_mode is selected_stages

agent_uidsstring[] required

Agents attached to the scorer; empty unless scope_mode is selected_agents

sampling_ratenumber double required

Percentage of the scorer's eligible runs that periodic scoring scores, to two decimal places. 0 stops automatic scoring.

model_idstring required

LLM model dispatched judge runs use to evaluate this scorer's rubric.

created_atstring date-time required

When the scorer was created

updated_atstring date-time required

When the scorer was updated

Changes

  • 4b5d145669beRevision changes
    • warning

      added the new selected_stages enum value to the response property for the response status

    • info

      added the new optional request property

    • info

      added the new selected_stages enum value to the request property

    • info

      added the required property to the response with the status

  • 7ee3ba55debcRevision changes
    • info

      api tag factory-scorers added

    • info

      api tag factory removed