Sign in
Retriever Benchmarks

Create benchmark

Create a new benchmark run to compare retriever pipelines. The benchmark will replay historical sessions and measure alignment with observed user behavior.

post/v1/retrievers/benchmarks

Request body

benchmark_namestring required

Human-readable name for this benchmark.

baseline_retriever_idstring required

ID of the baseline retriever pipeline to compare against.

candidate_retriever_idsstring[] required

IDs of candidate retriever pipelines to evaluate.

session_countinteger

Number of sessions to include in the benchmark.

Response

Successful Response

benchmark_idstring required

Unique benchmark identifier.

benchmark_namestring required

Human-readable name.

baseline_retriever_idstring required

Baseline retriever ID.

candidate_retriever_idsstring[] required

Candidate retriever IDs.

session_countinteger required

Number of sessions in benchmark.

status'pending' | 'building_sessions' | 'replaying' | 'computing_metrics' | 'completed' | 'failed' required

Status of a benchmark run.

comparisonBenchmarkComparison — unresolved $ref
created_atstring date-time required

Creation timestamp.

started_atstring date-time nullable

Execution start time.

completed_atstring date-time nullable

Completion time.

error_messagestring nullable

Error message if failed.

Changes

Changed in 1 of the 31 revisions of this API.1