Evaluations

Curate a dataset item from a generation

Promotes a real, completed generation into a test case: its input messages become the item's input, and its own answer becomes expected_output unless you supply one. Use it to build an evaluation set out of production traffic rather than hand-authoring fixtures.

The item is a copy, not a view. It keeps working after the source generation's content is purged, and source_generation_id goes null if that generation is deleted — a purge can never quietly stop a suite from being runnable.

Requires both evaluations:CreateDataset and generations:GetGeneration: the call copies content out of a generation, so a principal that may not read that generation may not curate it either.

Only a completed generation can be promoted (409 GENERATION_NOT_COMPLETED), and only while its content is still available: an agent or project running with trace_content_mode: none never stored the input, and a purged or expired generation no longer has it (409 GENERATION_CONTENT_UNAVAILABLE). Generations that predate input recording answer the same way.

post/v1/projects/{project_id}/datasets/{dataset_id}/items/from-generation

Request

  • Base URL: https://api.naturali.ai
  • URL: https://api.naturali.ai/v1/projects/{project_id}/datasets/{dataset_id}/items/from-generation
  • Auth: one of:
    • HTTP bearer
    • OAuth 2 (authorization code, authorization URL https://api.naturali.ai/authorize, token URL https://api.naturali.ai/token, scopes: mcp:access)

Path parameters

dataset_idstring required
Example:dset_V1StGXR8Z5jdHi6B

Dataset ID

Request body

generation_idstring required

The completed generation to promote. Must belong to the same project as the dataset.

expected_outputstring nullable

Reference answer. Omit to use the generation's own answer; pass null to store the item with no reference answer.

metadataNullableMetadataBag nullable

A MetadataBag on a request field where null is meaningful: a full-replacement update that clears the bag. A record whose bag is unset leaves the field out.

Example request

{
  "generation_id": "gen_V1StGXR8Z5jdHi6B",
  "expected_output": "Your invoice is issued on the first of each month.",
  "metadata": {
    "author": "John",
    "revision": 2
  }
}

Response

Dataset item created from the generation

idstring
dataset_idstring
expected_outputstring nullable
metadataMetadataBag

Caller-owned annotations on a resource, stored as the object they were written as: the types a value was written with are the types a read returns, so a filter can ask an ordering question about a number. Unlike other body fields, keys are stored and returned verbatim in the casing supplied — they are not converted between snake_case and camelCase.

No key is reserved, and that is the point: every piece of state the platform owns lives in its own typed column, so nothing written here reaches platform state. The platform never reads the bag — it is not an IAM context, not a policy input and not part of a prompt — which is what separates it from a tag bag.

source_generation_idstring nullable

The generation this item was curated from. A curated item is a deliberate fixture: erasing the source generation neither deletes nor mutates it.

created_atstring date-time
updated_atstring date-time

Example response

{
  "id": "dsit_V1StGXR8Z5jdHi6B",
  "dataset_id": "dset_V1StGXR8Z5jdHi6B",
  "input": [
    {
      "role": "user",
      "content": "When is my invoice issued?"
    }
  ],
  "metadata": {
    "author": "John",
    "revision": 2
  },
  "source_generation_id": "gen_V1StGXR8Z5jdHi6B"
}

Changes

No recorded changes to this operation across all 1 revision of this API.