Sign in
evaluators

Create an evaluator

Create a grader/evaluator. Current public evaluator types are llm, human, and code; legacy human-specific types remain readable for older evaluators. New clients should use the flat config fields (score_config, passing_conditions, llm_config, code_config) instead of relying on legacy configurations.

post/api/evaluators/

Headers

Authorizationstring required

Use your Respan API key for Respan API authentication. Enter only the Respan API key value; clients send Authorization: Bearer <RESPAN_API_KEY>. For /api/responses, OpenAI or Azure OpenAI provider credentials go in Settings -> Providers or the request body credential_override field, not in this auth field.

Request body

namestring required
evaluator_slugstring

Organization-scoped evaluator slug.

type'llm' | 'human' | 'code' required
score_value_type'numerical' | 'boolean' | 'percentage' | 'single_select' | 'multi_select' | 'text' | 'json' | 'comment' | 'categorical' required
eval_class'keywordsai_custom_llm' | 'custom_code'

Optional pre-built evaluator template.

descriptionstring nullable
passing_conditionsApiEvaluatorsPostRequestBodyContentApplicationJsonSchemaPassingConditions

Passing conditions in the standard Respan filter format.

configurationsobject

Legacy user-facing configuration object. New clients should prefer llm_config, code_config, score_config, and passing_conditions.

starredboolean

Response

Created evaluator.

idstring required

Stable evaluator ID shared by all versions.

version_idstring

Unique ID for this evaluator version.

versioninteger

Version number. The first draft is version 0.

is_read_onlyboolean

Whether this version is committed and immutable.

version_descriptionstring

Commit message for this version.

namestring required
evaluator_slugstring

Organization-scoped evaluator slug.

type'llm' | 'human' | 'code' required
score_value_type'numerical' | 'boolean' | 'percentage' | 'single_select' | 'multi_select' | 'text' | 'json' | 'comment' | 'categorical' required
eval_class'keywordsai_custom_llm' | 'custom_code'

Optional pre-built evaluator template.

descriptionstring nullable
passing_conditionsApiEvaluatorsPostResponsesContentApplicationJsonSchemaPassingConditions

Passing conditions in the standard Respan filter format.

configurationsobject

Legacy user-facing configuration object. New clients should prefer llm_config, code_config, score_config, and passing_conditions.

starredboolean
created_atstring date-time
updated_atstring date-time

Changes