Create an eval

Scorer configs, a discriminated union on type. Each type may appear at most once. Every scorer produces { score: 0–1, passed: boolean }; binary scorers emit 0 or 1.

exact_match compares the trimmed output text to expected_output. contains looks for value in the output text. json_logic evaluates expression over { input, output, object, expected, item.metadata }, where object is the structured output (absent when the agent has no output_schema). output_schema validates the structured output against the scorer's own schema, falling back to the agent's; it requires the agent to carry an output_schema, because without one the platform emits no structured output and every item would score 0.

llm_judge grades the output with a model completion, returning a continuous score plus its reasoning. Its pass_threshold is required: a continuous score says nothing about where "good enough" is, and a defaulted cutoff would silently decide the gate.

embedding_similarity embeds the output text and expected_output with the platform's configured embedding model (EMBEDDING_PROVIDER / EMBEDDING_MODEL — the same stack document ingestion uses) and scores their cosine similarity, clamped to 0-1. Its pass_threshold is required for the same reason as the judge's. An item without an expected_output scores 0; an embedding backend failure marks the item errored, never a score of 0.

tool runs a custom scoring algorithm: a server-callable project tool the engine invokes once per item with the item's context. Unlike the built-in types it may appear several times, each under a distinct name — outcomes and aggregates key on the name.

decider grades each item with a decision of a project decider: its score expression reads the decision's answers. Like tool, it keys on its name and may appear several times.

Scorer configs, a discriminated union on type. Each type may appear at most once. Every scorer produces { score: 0–1, passed: boolean }; binary scorers emit 0 or 1.

exact_match compares the trimmed output text to expected_output. contains looks for value in the output text. json_logic evaluates expression over { input, output, object, expected, item.metadata }, where object is the structured output (absent when the agent has no output_schema). output_schema validates the structured output against the scorer's own schema, falling back to the agent's; it requires the agent to carry an output_schema, because without one the platform emits no structured output and every item would score 0.

llm_judge grades the output with a model completion, returning a continuous score plus its reasoning. Its pass_threshold is required: a continuous score says nothing about where "good enough" is, and a defaulted cutoff would silently decide the gate.

embedding_similarity embeds the output text and expected_output with the platform's configured embedding model (EMBEDDING_PROVIDER / EMBEDDING_MODEL — the same stack document ingestion uses) and scores their cosine similarity, clamped to 0-1. Its pass_threshold is required for the same reason as the judge's. An item without an expected_output scores 0; an embedding backend failure marks the item errored, never a score of 0.

tool runs a custom scoring algorithm: a server-callable project tool the engine invokes once per item with the item's context. Unlike the built-in types it may appear several times, each under a distinct name — outcomes and aggregates key on the name.

decider grades each item with a decision of a project decider: its score expression reads the decision's answers. Like tool, it keys on its name and may appear several times.