Benchmarks

Start a benchmark run

Start an asynchronous benchmark run that evaluates extraction output against a ground truth dataset. The run is created with status queued; documents_total is set to the dataset's entry count and accuracy fields stay null until the run completes. Poll GET /v1/quality/benchmarks/{id} for progress and read per-document outcomes from GET /v1/quality/benchmarks/{id}/results.

post/v1/quality/benchmarks

Request

  • Base URL: https://api.talonic.com
  • URL: https://api.talonic.com/v1/quality/benchmarks
  • Auth: HTTP bearer

Request body

dataset_idstring uuid required

Ground truth dataset to benchmark against.

user_schema_idstring uuid required

User schema that defines the fields to evaluate.

namestring

Human-readable name for this run. Defaults to "Benchmark YYYY-MM-DD".

Example request

{
  "dataset_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "user_schema_id": "b2c3d4e5-f6a7-8901-bcde-f12345678901",
  "name": "Post-prompt-tune run"
}

Response

Benchmark run created with status queued.

idstring uuid required
namestring nullable
dataset_idstring uuid nullable
user_schema_idstring uuid nullable
statusstring required

Benchmark run status.

accuracy_overallnumber float nullable
accuracy_by_fieldobject nullable
documents_processedinteger nullable
documents_totalinteger nullable
duration_msinteger nullable
accuracy_deltanumber float nullable

Change in overall accuracy vs the compared run.

compared_to_run_idstring uuid nullable
created_atstring date-time
completed_atstring date-time nullable

Example response

{
  "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "dataset_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "user_schema_id": "b2c3d4e5-f6a7-8901-bcde-f12345678901",
  "status": "completed",
  "compared_to_run_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "created_at": "2026-04-25T14:30:00.000Z",
  "completed_at": "2026-04-25T14:30:00.000Z"
}

Changes

No recorded changes to this operation across all 1 revision of this API.