Create a new model evaluation using an existing benchmark.

This endpoint initiates a model evaluation using a benchmark from your library. Supports both single model evaluation and comparison mode where two models are evaluated side-by-side.

Request Body:

• model_id: Primary model to evaluate (required). It need not be deployed. A deployed model answers within seconds. An undeployed one is loaded by the GPU on demand and its answers wait for the next batch cycle, about 5 minutes.
• benchmark_id: ID of the benchmark to use for evaluation (required)
• baseline_model_id (optional): Baseline model ID for comparison mode
• custom_metrics (optional): defaults, the built-in metrics to run (binary_correctness, response_relevance; both run when omitted), and custom, up to 5 metrics of your own, each with name, prompt, parse_json, score_min and score_max. A custom name cannot reuse a built-in name. Each custom metric adds one judge call per question, so it adds time and cost. An invalid configuration is refused with 422
• evaluation_name (optional): Display name. Defaults to the evaluation id if omitted

Returns:

• evaluation_id: Unique identifier for tracking the evaluation
• model_id: The primary model being evaluated
• status: Initial status (always PROCESSING)
• benchmark_id: The benchmark being used
• created_at: Timestamp when evaluation was created
• message: Confirmation message

Example Request (Single Model):

POST /api/v3/evaluations/create
Headers: {"Authorization": "Bearer <api_key>"}

{
  "model_id": "model_01kmqm4nrn9fw6r",
  "benchmark_id": "benchmark_01k4x9m2p7q3r6a1",
  "custom_metrics": {
    "defaults": ["binary_correctness", "response_relevance"],
    "custom": [
      {
        "name": "hallucination_check",
        "prompt": "Does the generated answer contain factually incorrect information? Rate from 0-1.",
        "score_min": 0.0,
        "score_max": 1.0
      }
    ]
  }
}

Example Request (Comparison Mode):

POST /api/v3/evaluations/create
Headers: {"Authorization": "Bearer <api_key>"}

{
  "model_id": "Qwen/Qwen2.5-0.5B-Instruct",
  "baseline_model_id": "model_01kmqm4nrn9fw6r",
  "benchmark_id": "benchmark_01k4x9m2p7q3r6a1"
}

Example Response:

{
  "evaluation_id": "evaluation_01k4x9m2p7q3r7b1",
  "model_id": "model_01kmqm4nrn9fw6r",
  "status": "PROCESSING",
  "benchmark_id": "benchmark_01k4x9m2p7q3r6a1",
  "created_at": "2024-01-15T10:30:00Z",
  "message": "Evaluation created successfully and queued for execution"
}

Notes:

• Single model mode: Evaluates one model against the benchmark
• Comparison mode: Provide baseline_model_id to compare two models side-by-side
• Evaluation runs asynchronously - use the returned evaluation_id to check status
• Each metric reports <name>_mean, _min, _max, _std and _valid_count in the results, custom ones included. A _valid_count of 0 means the judge gave no usable score for that metric. Comparison mode adds the mean difference for custom metrics
• Use /evaluations/{evaluation_id}/status to track progress

{
  "model_id": "model_01k4x9m2p7q3r9d1",
  "benchmark_id": "benchmark_01k4x9m2p7q3r6a1",
  "evaluation_name": "Clause Review v2 Eval",
  "custom_metrics": {
    "defaults": [
      "binary_correctness",
      "response_relevance"
    ],
    "custom": [
      {
        "name": "hallucination_check",
        "prompt": "Does the generated answer contain factually incorrect information? Rate from 0-1."
      }
    ]
  },
  "baseline_model_id": "model_01k4x9m2p7q3r9d2"
}

{
  "evaluation_id": "evaluation_01k4x9m2p7q3r7b1",
  "evaluation_name": "Clause Review v2 Eval",
  "model_id": "model_01k4x9m2p7q3r9d1",
  "benchmark_id": "benchmark_01k4x9m2p7q3r6a1",
  "created_at": "2024-02-24T10:00:00Z",
  "message": "Evaluation created successfully"
}