gateway.openapi_Gateway

Create Evaluator

Creates a custom evaluator for scoring model outputs. Evaluators use the Eval Protocol to define test cases, run model inference, and score responses. They are used with evaluation jobs and Reinforcement Fine-Tuning (RFT).

Source Code Requirements

Your project should contain:

requirements.txt - Python dependencies for your evaluator
test_*.py - Pytest test file(s) with @evaluation_test decorated functions
Any additional code/modules your evaluator needs

Workflow

Recommended: Use the ep upload CLI command to handle all these steps automatically.

If using the API directly:

Call this endpoint to create the evaluator resource
Package your source directory as a .tar.gz (respecting .gitignore)
Call Get Evaluator Upload Endpoint to get a signed upload URL
PUT the tar.gz file to the signed URL
Call Validate Evaluator Upload to trigger server-side validation
Poll Get Evaluator until ready

Once active, reference the evaluator in Create Evaluation Job or Create Reinforcement Fine-tuning Job.

post/v1/accounts/{account_id}/evaluatorsV2

Path parameters

account_idstring required

The Account Id

Request body

evaluatorIdstring

Response

A successful response.

namestring

displayNamestring

descriptionstring

createTimestring date-time

createdBystring

updateTimestring date-time

state'STATE_UNSPECIFIED' | 'ACTIVE' | 'BUILDING' | 'BUILD_FAILED'

requirementsstring

entryPointstring

commitHashstring

defaultDatasetstring

Changes

Changed in 6 of the 27 revisions of this API.912

Jan 30, 2026920bcfb416a72See the full diff
- ●
  removed the request property /
  request-property-removed
- ●
  removed the optional property from the response with the status
  response-optional-property-removed
This revision also has 5 changes that name no endpoint, such as unreferenced schemas being removed. See the revision's changelog
Jan 13, 2026e9e5bbd5fb9d2See the full diff
- ○
  added the new optional request property /
  new-optional-request-property
- ○
  added the optional property to the response with the status
  response-optional-property-added
Jan 12, 20269823da1648372See the full diff
- ○
  added the new optional request property /
  new-optional-request-property
- ○
  added the optional property to the response with the status
  response-optional-property-added
This revision also has 1 change that name no endpoint, such as unreferenced schemas being removed. See the revision's changelog
Dec 23, 2025d8f81c17e7426See the full diff
- ●
  removed the request property /
  request-property-removed
- ●
  removed the request property /
  request-property-removed
- ●
  removed the request property /
  request-property-removed
- ●
  removed the optional property from the response with the status
  response-optional-property-removed
- ●
  removed the optional property from the response with the status
  response-optional-property-removed
- ●
  removed the optional property from the response with the status
  response-optional-property-removed
This revision also has 4 changes that name no endpoint, such as unreferenced schemas being removed. See the revision's changelog
Nov 13, 2025cf095a16263f11See the full diff
- ●
  added the new TYPE_TEMPORARY enum value to the / response property for the response status
  response-property-enum-value-added
- ○
  added the new TYPE_TEMPORARY enum value to the request property //
  request-property-enum-value-added
This revision also has 1 change that name no endpoint, such as unreferenced schemas being removed. See the revision's changelog
Nov 6, 2025311efdb2fa577See the full diff
- ○
  the endpoint scheme security BearerAuth was added to the API
  api-security-added
- ○
  api tag gateway.openapi_Gateway added
  api-tag-added
- ○
  api tag Gateway removed
  api-tag-removed
- ○
  added the new optional request property /
  new-optional-request-property
- ○
  added the new optional request property /
  new-optional-request-property
- ○
  added the optional property to the response with the status
  response-optional-property-added
- ○
  added the optional property to the response with the status
  response-optional-property-added
This revision also has 2 changes that name no endpoint, such as unreferenced schemas being removed. See the revision's changelog

Full history