Sign in
Evaluators

Create evaluator

Creates a new evaluator with an initial version.

Payload Requirements

  • The evaluator name must be unique within the given space.
  • type (top-level) selects the evaluator kind: TEMPLATE or CODE. With TEMPLATE, provide version.template_config. With CODE, provide version.code_config — where code_config.type is MANAGED or CUSTOM (a separate discriminator within code_config, independent of the top-level type: CODE).
  • For template evaluators: version.template_config.name is the eval column name; must match ^[a-zA-Z0-9_\s\-&()]+$.
  • For template evaluators: version.template_config.template is the prompt template; use {variable} for placeholders (f-string format, e.g. {input}, {output}).
  • For template evaluators: version.template_config.classification_choices is required and maps choice labels to numeric scores (e.g. {"relevant": 1, "irrelevant": 0}).
  • For code evaluators: see CodeConfig — managed evaluators (code_config.type: MANAGED) use managed_evaluator and variables; custom evaluators (code_config.type: CUSTOM) use code, optional imports, and variables.
  • System-managed fields (id, created_at, updated_at, created_by_user_id) are rejected on input.

Valid example (template evaluator)

{
  "name": "Hallucination Detector",
  "space_id": "U3BhY2U6MTpWNEth",
  "type": "TEMPLATE",
  "version": {
    "commit_message": "Initial version",
    "template_config": {
      "name": "hallucination",
      "template": "Given the input: {input}\nand the output: {output}\nIs the output a hallucination?",
      "include_explanations": true,
      "use_function_calling": true,
      "classification_choices": {"hallucinated": 0, "factual": 1},
      "llm_config": {
        "ai_integration_id": "TGxtSW50ZWdyYXRpb246MTI6YUJjRA==",
        "model_name": "gpt-4o",
        "invocation_parameters": {"temperature": 0},
        "provider_parameters": {}
      }
    }
  }
}

Invalid example (type/config mismatch — TEMPLATE type with code_config)

{
  "name": "Bad Evaluator",
  "space_id": "U3BhY2U6MTpWNEth",
  "type": "TEMPLATE",
  "version": {
    "commit_message": "Wrong config",
    "code_config": {
      "type": "CUSTOM",
      "name": "my_eval",
      "code": "class Evaluator: ...",
      "variables": ["input"]
    }
  }
}

<Note>This endpoint is in beta, read more here.</Note>

post/v2/evaluators

Request body

space_idstring required

Space identifier (base64)

namestring required

Evaluator name (must be unique within the space)

descriptionstring

Evaluator description

type'TEMPLATE' | 'CODE' | 'HARNESS' | 'REMOTE' required

The evaluator type:

  • TEMPLATE — LLM-based evaluator.
  • CODE — managed built-in evaluators or custom Python code (both are subtypes of CODE, discriminated by the nested CodeConfig.type = MANAGED | CUSTOM).
  • HARNESS — test harness evaluator.
  • REMOTE — remote evaluator.

Applies to both the parent Evaluator.type field and every version's type discriminator — a version's type must always match its parent evaluator's type.

Response

Returns an evaluator with a resolved version

idstring required

The unique identifier for the evaluator

namestring required

The name of the evaluator

descriptionstring nullable

The description of the evaluator

type'TEMPLATE' | 'CODE' | 'HARNESS' | 'REMOTE' required

The evaluator type:

  • TEMPLATE — LLM-based evaluator.
  • CODE — managed built-in evaluators or custom Python code (both are subtypes of CODE, discriminated by the nested CodeConfig.type = MANAGED | CUSTOM).
  • HARNESS — test harness evaluator.
  • REMOTE — remote evaluator.

Applies to both the parent Evaluator.type field and every version's type discriminator — a version's type must always match its parent evaluator's type.

space_idstring required

The unique identifier for the space the evaluator belongs to

created_atstring date-time required

When the evaluator was created

updated_atstring date-time required

When the evaluator was last updated

created_by_user_idstring nullable required

The unique identifier for the user who created the evaluator

Changes

Changed in 4 of the 16 revisions of this API.867

  • 9263b98aaefb1See the full diff
    • ○

      added the optional property allOf[subschema #2]/version/oneOf[EvaluatorVersionTemplate]/allOf[subschema #2]/template_config/llm_config/invocation_parameters/service_tier to the response with the 201 status

      response-optional-property-added

  • c26a00e9b318211See the full diff
    • ▲

      added the new required request property version/oneOf[CreateTemplateEvaluatorVersionRequest]/template_config/use_function_calling

      new-required-request-property

    • ▲

      removed the required property allOf[subschema #2]/version/oneOf[EvaluatorVersionTemplate]/allOf[subschema #2]/template_config/use_function_calling_if_available from the response with the 201 status

      response-required-property-removed

    • ●

      removed the request property version/oneOf[CreateTemplateEvaluatorVersionRequest]/template_config/use_function_calling_if_available

      request-property-removed

    • ○

      added the optional property allOf[subschema #2]/version/oneOf[EvaluatorVersionTemplate]/allOf[subschema #2]/template_config/use_function_calling to the response with the 201 status

      response-optional-property-added

  • 4d95080d8ffb11See the full diff
    • ▲

      added ToolConfigRequest to the version/oneOf[CreateTemplateEvaluatorVersionRequest]/template_config/llm_config/invocation_parameters/tool_config request property allOf list

      request-property-all-of-added

    • ●

      removed ToolConfig from the version/oneOf[CreateTemplateEvaluatorVersionRequest]/template_config/llm_config/invocation_parameters/tool_config request property allOf list

      request-property-all-of-removed

  • 1e87d8a4cf69545See the full diff
    • ▲

      added AnthropicHeadersRequest to the version/oneOf[CreateTemplateEvaluatorVersionRequest]/template_config/llm_config/provider_parameters/anthropic_headers request property allOf list

      request-property-all-of-added

    • ▲

      added AzureParamsRequest to the version/oneOf[CreateTemplateEvaluatorVersionRequest]/template_config/llm_config/provider_parameters/azure_params request property allOf list

      request-property-all-of-added

    • ▲

      added BedrockOptionsRequest to the version/oneOf[CreateTemplateEvaluatorVersionRequest]/template_config/llm_config/provider_parameters/bedrock_options request property allOf list

      request-property-all-of-added

    • ▲

      added ResponseFormatRequest to the version/oneOf[CreateTemplateEvaluatorVersionRequest]/template_config/llm_config/invocation_parameters/response_format request property allOf list

      request-property-all-of-added

    • ▲

      removed ManagedCodeConfig CustomCodeConfig from the version/oneOf[CreateCodeEvaluatorVersionRequest]/code_config request property oneOf list

      request-property-one-of-removed

    • ●

      removed AnthropicHeaders from the version/oneOf[CreateTemplateEvaluatorVersionRequest]/template_config/llm_config/provider_parameters/anthropic_headers request property allOf list

      request-property-all-of-removed

    • ●

      removed AzureParams from the version/oneOf[CreateTemplateEvaluatorVersionRequest]/template_config/llm_config/provider_parameters/azure_params request property allOf list

      request-property-all-of-removed

    • ●

      removed BedrockOptions from the version/oneOf[CreateTemplateEvaluatorVersionRequest]/template_config/llm_config/provider_parameters/bedrock_options request property allOf list

      request-property-all-of-removed

    • ●

      removed ResponseFormat from the version/oneOf[CreateTemplateEvaluatorVersionRequest]/template_config/llm_config/invocation_parameters/response_format request property allOf list

      request-property-all-of-removed

    • ○

      the request property version/oneOf[CreateTemplateEvaluatorVersionRequest]/template_config/classification_choices became nullable

      request-property-became-nullable

    • ○

      the request property version/oneOf[CreateTemplateEvaluatorVersionRequest]/template_config/classification_choices became optional

      request-property-became-optional

    • ○

      mapped value for discriminator key CUSTOM changed from CustomCodeConfig to CustomCodeConfigRequest for version/oneOf[CreateCodeEvaluatorVersionRequest]/code_config request property

      request-property-discriminator-mapping-changed

    • ○

      mapped value for discriminator key MANAGED changed from ManagedCodeConfig to ManagedCodeConfigRequest for version/oneOf[CreateCodeEvaluatorVersionRequest]/code_config request property

      request-property-discriminator-mapping-changed

    • ○

      added ManagedCodeConfigRequest CustomCodeConfigRequest to the version/oneOf[CreateCodeEvaluatorVersionRequest]/code_config request property oneOf list

      request-property-one-of-added

Of the 16 revisions, 1 has no diff computed.