Sign in
Tasks

Create task

Creates a new task. Supported task types:

typeData sourceNotes
TEMPLATE_EVALUATIONproject_id or dataset_idRequires evaluators. Supports continuous operation.
CODE_EVALUATIONproject_id or dataset_idRequires evaluators. Supports continuous operation.
RUN_EXPERIMENTdataset_id onlyRequires run_configuration. Never continuous.

For RUN_EXPERIMENT tasks the run configuration is stored on the task. Each trigger (POST /v2/tasks/{task_id}/trigger) supplies per-run fields (experiment_name, optional example subset, etc.) and starts an async run. Poll GET /v2/task-runs/{run_id} until status reaches a terminal state.

Payload Requirements (template_evaluation / code_evaluation)

  • At least one evaluator is required.
  • Duplicate evaluator IDs are not allowed.
  • When dataset_id is provided, experiment_ids must contain at least one entry.
  • sampling_rate and is_continuous are only supported on project-based tasks.
  • System-managed fields (id, created_at, updated_at) are rejected on input.
  • evaluator_version_id pins an evaluator to one version. Omit it (or send null) to run that evaluator's latest version, which is the default. The version must belong to the evaluator named by evaluator_id, and every evaluator on the task must resolve to the same data scope — both return 422. List an evaluator's versions with GET /v2/evaluators/{evaluator_id}/versions.

Payload Requirements (run_experiment)

  • dataset_id is required; project_id must be omitted.
  • run_configuration is required; evaluators, experiment_ids, sampling_rate, is_continuous, and query_filter must be omitted.

Valid example (template_evaluation, project-based)

{
  "name": "Production Hallucination Check",
  "type": "TEMPLATE_EVALUATION",
  "project_id": "TW9kZWw6MTIzOmFCY0Q=",
  "sampling_rate": 1.0,
  "is_continuous": true,
  "evaluators": [
    {
      "evaluator_id": "RXZhbHVhdG9yOjEyOmFCY0Q=",
      "column_mappings": {"input": "attributes.input.value", "output": "attributes.output.value"}
    }
  ]
}

Valid example (pinned to a specific evaluator version)

{
  "name": "Hallucination Check v3",
  "type": "TEMPLATE_EVALUATION",
  "project_id": "TW9kZWw6MTIzOmFCY0Q=",
  "evaluators": [
    {
      "evaluator_id": "RXZhbHVhdG9yOjEyOmFCY0Q=",
      "evaluator_version_id": "RXZhbHVhdG9yVmVyc2lvbjo5OTphQmNE",
      "column_mappings": {"input": "attributes.input.value"}
    }
  ]
}

Invalid example (run_experiment missing run_configuration)

{
  "name": "My Experiment",
  "type": "RUN_EXPERIMENT",
  "dataset_id": "RGF0YXNldDo1NjpxUndY"
}

Invalid example (422 — the version belongs to a different evaluator)

{
  "name": "Mismatched Pin",
  "type": "TEMPLATE_EVALUATION",
  "project_id": "TW9kZWw6MTIzOmFCY0Q=",
  "evaluators": [
    {
      "evaluator_id": "RXZhbHVhdG9yOjEyOmFCY0Q=",
      "evaluator_version_id": "RXZhbHVhdG9yVmVyc2lvbjo3OmFCY0Q="
    }
  ]
}

<Note>This endpoint is in beta, read more here.</Note>

post/v2/tasks

Request body

OR
OR

Response

Returns a single task object

idstring required

The unique identifier for the task

namestring required

The name of the task

type'TEMPLATE_EVALUATION' | 'CODE_EVALUATION' | 'RUN_EXPERIMENT' required

The task type.

  • TEMPLATE_EVALUATION - An LLM template-based evaluation task.
  • CODE_EVALUATION - A code-based evaluation task.
  • RUN_EXPERIMENT - A task that runs experiments.
project_idstring nullable

The project identifier (base64). Present for project-based tasks.

dataset_idstring nullable

The dataset identifier (base64). Present for dataset-based tasks.

sampling_ratenumber nullable

Sampling rate between 0 and 1. Only applicable for project-based tasks.

is_continuousboolean required

Whether the task runs continuously on incoming data.

query_filterstring nullable required

Task-level query filter applied to all data. Span-granularity shape only. Null when the task uses the trace/session shape (query_filters). Mutually exclusive with query_filters.

experiment_idsstring[] required

Experiment identifiers (base64) for dataset-based tasks.

last_run_atstring date-time nullable required

When the task was last run.

created_atstring date-time required

When the task was created.

updated_atstring date-time required

When the task was last updated.

created_by_user_idstring nullable required

The unique identifier for the user who created the task.

Changes

Changed in 3 of the 16 revisions of this API.3812

  • f63ac334fd4a286See the full diff
    • ▲

      the oneOf[CreateCodeEvaluationTaskRequest]/evaluators/items/ request property type/format changed from object/ to /

      request-property-type-changed

    • ▲

      the oneOf[CreateTemplateEvaluationTaskRequest]/evaluators/items/ request property type/format changed from object/ to /

      request-property-type-changed

    • ●

      removed the request property oneOf[CreateCodeEvaluationTaskRequest]/evaluators/items/column_mappings

      request-property-removed

    • ●

      removed the request property oneOf[CreateCodeEvaluationTaskRequest]/evaluators/items/evaluator_id

      request-property-removed

    • ●

      removed the request property oneOf[CreateCodeEvaluationTaskRequest]/evaluators/items/evaluator_version_id

      request-property-removed

    • ●

      removed the request property oneOf[CreateCodeEvaluationTaskRequest]/evaluators/items/query_filter

      request-property-removed

    • ●

      removed the request property oneOf[CreateTemplateEvaluationTaskRequest]/evaluators/items/column_mappings

      request-property-removed

    • ●

      removed the request property oneOf[CreateTemplateEvaluationTaskRequest]/evaluators/items/evaluator_id

      request-property-removed

    • ●

      removed the request property oneOf[CreateTemplateEvaluationTaskRequest]/evaluators/items/evaluator_version_id

      request-property-removed

    • ●

      removed the request property oneOf[CreateTemplateEvaluationTaskRequest]/evaluators/items/query_filter

      request-property-removed

    • ○

      added the new optional request property oneOf[CreateCodeEvaluationTaskRequest]/query_filters

      new-optional-request-property

    • ○

      added the new optional request property oneOf[CreateTemplateEvaluationTaskRequest]/query_filters

      new-optional-request-property

    • ○

      added SpanEvaluatorInput TraceOrSessionEvaluatorInput to the oneOf[CreateCodeEvaluationTaskRequest]/evaluators/items/ request property oneOf list

      request-property-one-of-added

    • ○

      added SpanEvaluatorInput TraceOrSessionEvaluatorInput to the oneOf[CreateTemplateEvaluationTaskRequest]/evaluators/items/ request property oneOf list

      request-property-one-of-added

    • ○

      added the optional property evaluators/items/query_mappings to the response with the 201 status

      response-optional-property-added

    • ○

      added the optional property query_filters to the response with the 201 status

      response-optional-property-added

  • 9263b98aaefb2See the full diff
    • ○

      added the optional property run_configuration/allOf[RunConfiguration]/oneOf[LlmGenerationRunConfig]/invocation_parameters/service_tier to the response with the 201 status

      response-optional-property-added

    • ○

      added the optional property run_configuration/allOf[RunConfiguration]/oneOf[TemplateEvaluationRunConfig]/invocation_parameters/service_tier to the response with the 201 status

      response-optional-property-added

  • 4d95080d8ffb14See the full diff
    • ▲

      removed LlmGenerationRunConfig TemplateEvaluationRunConfig AgentCallRunConfig from the oneOf[CreateRunExperimentTaskRequest]/run_configuration request property oneOf list

      request-property-one-of-removed

    • ○

      mapped value for discriminator key AGENT_CALL changed from AgentCallRunConfig to AgentCallRunConfigRequest for oneOf[CreateRunExperimentTaskRequest]/run_configuration request property

      request-property-discriminator-mapping-changed

    • ○

      mapped value for discriminator key LLM_GENERATION changed from LlmGenerationRunConfig to LlmGenerationRunConfigRequest for oneOf[CreateRunExperimentTaskRequest]/run_configuration request property

      request-property-discriminator-mapping-changed

    • ○

      mapped value for discriminator key TEMPLATE_EVALUATION changed from TemplateEvaluationRunConfig to TemplateEvaluationRunConfigRequest for oneOf[CreateRunExperimentTaskRequest]/run_configuration request property

      request-property-discriminator-mapping-changed

    • ○

      added LlmGenerationRunConfigRequest TemplateEvaluationRunConfigRequest AgentCallRunConfigRequest to the oneOf[CreateRunExperimentTaskRequest]/run_configuration request property oneOf list

      request-property-one-of-added

Of the 16 revisions, 1 has no diff computed.