---
title: "Create Evaluation"
method: POST
path: "/api/v3/evaluations/create"
tags: ["Evaluations"]
---

# Create Evaluation

`POST /api/v3/evaluations/create`

## Request

- Base URL: `https://api.nugen.in`
- URL: `https://api.nugen.in/api/v3/evaluations/create`
- Auth: HTTP bearer

Create a new model evaluation using an existing benchmark.

This endpoint initiates a model evaluation using a benchmark from your library. Supports both single model evaluation and comparison mode where two models are evaluated side-by-side.

**Request Body:**

- `model_id`: Primary model to evaluate (required). It need not be deployed. A deployed model answers within seconds. An undeployed one is loaded by the GPU on demand and its answers wait for the next batch cycle, about 5 minutes.
- `benchmark_id`: ID of the benchmark to use for evaluation (required)
- `baseline_model_id` (optional): Baseline model ID for comparison mode
- `custom_metrics` (optional): `defaults`, the built-in metrics to run (`binary_correctness`, `response_relevance`; both run when omitted), and `custom`, up to 5 metrics of your own, each with `name`, `prompt`, `parse_json`, `score_min` and `score_max`. A custom `name` cannot reuse a built-in name. Each custom metric adds one judge call per question, so it adds time and cost. An invalid configuration is refused with 422
- `evaluation_name` (optional): Display name. Defaults to the evaluation id if omitted

**Returns:**

- `evaluation_id`: Unique identifier for tracking the evaluation
- `model_id`: The primary model being evaluated
- `status`: Initial status (always `PROCESSING`)
- `benchmark_id`: The benchmark being used
- `created_at`: Timestamp when evaluation was created
- `message`: Confirmation message

**Example Request (Single Model):**

```json
POST /api/v3/evaluations/create
Headers: {"Authorization": "Bearer <api_key>"}

{
  "model_id": "model_01kmqm4nrn9fw6r",
  "benchmark_id": "benchmark_01k4x9m2p7q3r6a1",
  "custom_metrics": {
    "defaults": ["binary_correctness", "response_relevance"],
    "custom": [
      {
        "name": "hallucination_check",
        "prompt": "Does the generated answer contain factually incorrect information? Rate from 0-1.",
        "score_min": 0.0,
        "score_max": 1.0
      }
    ]
  }
}
```

**Example Request (Comparison Mode):**

```json
POST /api/v3/evaluations/create
Headers: {"Authorization": "Bearer <api_key>"}

{
  "model_id": "Qwen/Qwen2.5-0.5B-Instruct",
  "baseline_model_id": "model_01kmqm4nrn9fw6r",
  "benchmark_id": "benchmark_01k4x9m2p7q3r6a1"
}
```

**Example Response:**

```json
{
  "evaluation_id": "evaluation_01k4x9m2p7q3r7b1",
  "model_id": "model_01kmqm4nrn9fw6r",
  "status": "PROCESSING",
  "benchmark_id": "benchmark_01k4x9m2p7q3r6a1",
  "created_at": "2024-01-15T10:30:00Z",
  "message": "Evaluation created successfully and queued for execution"
}
```
**Notes:**

- Single model mode: Evaluates one model against the benchmark
- Comparison mode: Provide `baseline_model_id` to compare two models side-by-side
- Evaluation runs asynchronously - use the returned `evaluation_id` to check status
- Each metric reports `<name>_mean`, `_min`, `_max`, `_std` and `_valid_count` in the results, custom ones included. A `_valid_count` of 0 means the judge gave no usable score for that metric. Comparison mode adds the mean difference for custom metrics
- Use `/evaluations/{evaluation_id}/status` to track progress

## Request body

- EvaluationCreateRequest: Start an evaluation of a model, or a comparison of two models, on a benchmark.
  - `model_id` string, required: ID of the model to evaluate
  - `benchmark_id` string, required: ID of an existing benchmark
  - `evaluation_name` string, nullable: Display name for the evaluation. Defaults to the evaluation id if omitted.
  - `custom_metrics` EvaluationMetricsConfig: The default and custom metrics an evaluation uses.
    - `defaults` string[]: Built-in metrics to run. Both run when omitted.
    - `custom` EvaluationCustomMetric[]: Your own metrics, at most 5. Each adds one judge call per question.
      - `name` string, required: Lowercase letters, digits and underscores, starting with a letter. Names the score in the results.
      - `prompt` string, required: What the judge scores for each answer.
      - `parse_json` boolean: Read the judge reply as JSON.
      - `score_min` number: Lowest score the judge can give.
      - `score_max` number: Highest score the judge can give.
  - `baseline_model_id` string, nullable: ID of baseline model for comparison mode (eval-compare)

## Response `200`

Returns a unique identifier for the initiated evaluation along with the initial evaluation status. This endpoint starts an asynchronous evaluation using a specified benchmark and model(s), allowing users to track progress and retrieve results once completed.

- EvaluationCreateResponse: Acknowledgement of a new evaluation.
  - `evaluation_id` string, required: Unique identifier for the evaluation
  - `evaluation_name` string, required: Display name of the evaluation
  - `model_id` string, required: ID of the model being evaluated
  - `status` 'QUEUED' | 'PROCESSING' | 'READY' | 'FAILED' | 'STOPPED', required: Lifecycle status of a resource that runs asynchronously. Stored as is and returned as is, with no translation in between. - `QUEUED`: accepted and waiting for its workflow to start - `PROCESSING`: running - `READY`: finished successfully - `FAILED`: failed at any stage - `STOPPED`: stopped by the user before completion
  - `benchmark_id` string, required: Benchmark ID used
  - `created_at` string, date-time, required: timestamp of evaluation creation
  - `message` string, required: Status message

## Other responses

- `401`: Invalid API key
- `403`: Authorization header missing or not a Bearer token
- `422`: Validation Error

---

[API](https://skmtc.dev/nugen/apis/nugen-intelligence-api.md) - [All operations](https://skmtc.dev/nugen/apis/nugen-intelligence-api/llms.txt) - [OpenAPI document](https://skmtc.dev/nugen/apis/nugen-intelligence-api/revisions/62baa7960a4a?raw)