---
title: "Create or update a custom evaluator configuration"
method: PUT
path: "/api/unstable/llm-obs/config/evaluators/custom/{eval_name}"
tags: ["LLM Observability"]
---

# Create or update a custom evaluator configuration

`PUT /api/unstable/llm-obs/config/evaluators/custom/{eval_name}`

## Request

- Base URL: `https://api.datadoghq.com`
- URL: `https://api.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name}`
- Auth: API key in header `DD-API-KEY` and API key in header `DD-APPLICATION-KEY`

Create or update a custom LLM Observability evaluator configuration by its name.

## Path parameters

- `eval_name` string, required

## Request body

- LLMObsCustomEvalConfigUpdateRequest: Request to create or update a custom LLM Observability evaluator configuration.
  - `data` LLMObsCustomEvalConfigUpdateData, required: Data object for creating or updating a custom LLM Observability evaluator configuration.
    - `attributes` LLMObsCustomEvalConfigUpdateAttributes, required: Attributes for creating or updating a custom LLM Observability evaluator configuration.
      - `category` string: Category of the evaluator.
      - `eval_name` string: Name of the custom evaluator. If provided, must match the eval_name path parameter.
      - `llm_judge_config` LLMObsCustomEvalConfigLLMJudgeConfig: LLM judge configuration for a custom evaluator.
        - `assessment_criteria` LLMObsCustomEvalConfigAssessmentCriteria: Criteria used to assess the pass/fail result of a custom evaluator.
          - `max_threshold` number, double, nullable: Maximum numeric threshold for a passing result.
          - `min_threshold` number, double, nullable: Minimum numeric threshold for a passing result.
          - `pass_values` string[], nullable: Specific output values considered as a passing result.
          - `pass_when` boolean, nullable: When true, a boolean output of true is treated as passing.
        - `inference_params` LLMObsCustomEvalConfigInferenceParams, required: LLM inference parameters for a custom evaluator.
          - `frequency_penalty` number, double: Frequency penalty to reduce repetition.
          - `max_tokens` integer: Maximum number of tokens to generate.
          - `presence_penalty` number, double: Presence penalty to reduce repetition.
          - `temperature` number, double: Sampling temperature for the LLM.
          - `top_k` integer: Top-k sampling parameter.
          - `top_p` number, double: Top-p (nucleus) sampling parameter.
        - `last_used_library_prompt_template_name` string, nullable: Name of the last library prompt template used.
        - `modified_library_prompt_template` boolean, nullable: Whether the library prompt template was modified.
        - `output_schema` object, nullable: JSON schema describing the expected output format of the LLM judge.
        - `parsing_type` 'structured_output' | 'json': Output parsing type for a custom LLM judge evaluator.
        - `prompt_template` LLMObsCustomEvalConfigPromptMessage[]: List of messages forming the LLM judge prompt template.
          - `content` string: Text content of the message.
          - `contents` LLMObsCustomEvalConfigPromptContent[]: Multi-part content blocks for the message.
            - `type` string, required: Content block type.
            - `value` LLMObsCustomEvalConfigPromptContentValue, required: Value of a prompt message content block.
              - …
          - `role` string, required: Role of the message author.
      - `llm_provider` LLMObsCustomEvalConfigLLMProvider: LLM provider configuration for a custom evaluator.
        - `bedrock` LLMObsCustomEvalConfigBedrockOptions: AWS Bedrock-specific options for LLM provider configuration.
          - `region` string: AWS region for Bedrock.
        - `integration_account_id` string: Integration account identifier.
        - `integration_provider` 'openai' | 'amazon-bedrock' | 'anthropic' | 'azure-openai' | 'vertex-ai' | 'llm-proxy': Name of the LLM integration provider.
        - `model_name` string: Name of the LLM model.
        - `vertex_ai` LLMObsCustomEvalConfigVertexAIOptions: Google Vertex AI-specific options for LLM provider configuration.
          - `location` string: Google Cloud region.
          - `project` string: Google Cloud project ID.
      - `target` LLMObsCustomEvalConfigTarget, required: Target application configuration for a custom evaluator.
        - `application_name` string, required: Name of the ML application this evaluator targets.
        - `enabled` boolean, required: Whether the evaluator is active for the target application.
        - `eval_scope` 'span' | 'trace' | 'session': Scope at which to evaluate spans.
        - `filter` string, nullable: Filter expression to select which spans to evaluate.
        - `root_spans_only` boolean, nullable: When true, only root spans are evaluated.
        - `sampling_percentage` number, double, nullable: Percentage of traces to evaluate. Must be greater than 0 and at most 100.
    - `id` string: Name of the evaluator. If provided, must match the eval_name path parameter.
    - `type` 'evaluator_config', required: Type of the custom LLM Observability evaluator configuration resource.

## Response `200`

OK

## Other responses

- `400`: Bad Request
- `401`: Unauthorized
- `403`: Forbidden
- `404`: Not Found
- `422`: Unprocessable Entity
- `429`: Too many requests

---

[API](https://skmtc.dev/datadog/apis/datadog-api-v2-collection.md) - [All operations](https://skmtc.dev/datadog/apis/datadog-api-v2-collection/llms.txt) - [OpenAPI document](https://skmtc.dev/datadog/apis/datadog-api-v2-collection/revisions/da68bf029e4c?raw)