ADadaptionlabs
datasets

Start a custom-rubric evaluation for an adapted dataset

Changed on
post/api/v1/datasets/{dataset_id}/custom-evals

Request

Path parameters

dataset_idstring required

Request body

judge_promptstring required

The judge rubric. Scored per sample against the original and adapted pairs.

namestring

Optional label so a history of rubrics stays legible.

Example request

{
  "judge_prompt": "Rate how factually accurate the response is, from 0 (fabricated) to 10 (fully supported)."
}

Response

custom_eval_idstring required
dataset_idstring required
statusstring required

pending | running | succeeded | failed

judge_promptstring required
namestring nullable required
score_beforenumber nullable required

Mean rubric score on the original pairs. Null until the eval succeeds.

score_afternumber nullable required

Mean rubric score on the adapted pairs. Null until the eval succeeds.

improvement_percentnumber nullable required

Percentage change from score_before to score_after. Null until the eval succeeds.

scored_rowsinteger nullable required

Rows the judge scored. A rubric score is an average over this sample, not the whole dataset.

error_messagestring nullable required
created_atstring date-time required
completed_atstring date-time nullable required

Example response

{
  "status": "running"
}

Changes