Sign in
evals

Create Run

Create a new run for an evaluation.

Model-based routing (for multi-account support):

  • Pass model via header: x-litellm-model: gpt-4-account-1
  • Pass model via query: ?model=gpt-4-account-1
  • Pass model via body: {"model": "gpt-4-account-1"}
  • Pass model via completion.model: {"completion": {"model": "gpt-4-account-1"}}

Example usage:

curl -X POST "http://localhost:4000/v1/evals/eval_123/runs"       -H "Authorization: Bearer your-key"       -H "Content-Type: application/json"       -d '{
    "data_source": {"type": "dataset", "dataset_id": "dataset_123"},
    "completion": {"model": "gpt-4", "temperature": 0.7}
  }'

Returns: Run object with id, status, timestamps, etc.

post/v1/evals/{eval_id}/runs

Path parameters

eval_idstring required

Query parameters

custom_llm_providerstring nullable

Response

Successful Response

completed_atinteger nullable
created_atinteger required
data_sourceobject required
errorobject nullable
eval_idstring required
idstring required
metadataobject nullable
modelstring nullable
namestring nullable
objectstring
{"stackTrail":"components:schemas:Run:properties:per_model_usage:anyOf","oasType":"schema","type":"unknown","title":"Per Model Usage","nullable":true}
report_urlstring nullable
result_countsobject nullable
shared_with_openaiboolean nullable
started_atinteger nullable
status'queued' | 'running' | 'completed' | 'failed' | 'cancelled' required

Changes

Changed in 2 of the 42 revisions of this API.6

  • c357baf422ab2See the full diff
    • ○

      api tag evals added

      api-tag-added

    • ○

      api tag OpenAI Evals API - Runs removed

      api-tag-removed

  • 082b5fabd9094See the full diff
    • ○

      api tag OpenAI Evals API - Runs added

      api-tag-added

    • ○

      api tag evals removed

      api-tag-removed

    • ○

      added the optional property // to the response with the status

      response-optional-property-added

    • ○

      added the optional property // to the response with the status

      response-optional-property-added

    This revision also has 5 changes that name no endpoint, such as unreferenced schemas being removed. See the revision's changelog