---
title: "List custom evaluator configurations"
method: GET
path: "/api/unstable/llm-obs/config/evaluators/custom"
tags: ["Agent Observability"]
---

# List custom evaluator configurations

`GET /api/unstable/llm-obs/config/evaluators/custom`

## Request

- Base URL: `https://api.datadoghq.com`
- URL: `https://api.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom`
- Auth: one of:
  - API key in header `DD-API-KEY` and API key in header `DD-APPLICATION-KEY`
  - OAuth 2
    - Authorization code: `/oauth2/v1/authorize`
    - Token URL: `/oauth2/v1/token`
- TypeScript: <https://skmtc.dev/datadog/apis/datadog-api-v2-collection/docs/api/unstable/llm-obs/config/evaluators/custom/get.ts> (a function with typed inputs; its response is checked against the schema)

List all custom Agent Observability evaluator configurations for the organization.

## Response `200`

OK

- LLMObsCustomEvalConfigListResponse: Response containing a list of custom Agent Observability evaluator configurations.
  - `data` LLMObsCustomEvalConfigData[], required: List of custom evaluator configuration data objects.
    - `attributes` LLMObsCustomEvalConfigAttributes, required: Attributes of a custom Agent Observability evaluator configuration.
      - `category` string: Category of the evaluator.
      - `created_at` string, date-time, required: Timestamp when the evaluator configuration was created.
      - `created_by` LLMObsCustomEvalConfigUser: A Datadog user associated with a custom evaluator configuration.
        - `email` string: Email address of the user.
      - `eval_name` string, required: Name of the custom evaluator.
      - `last_updated_by` LLMObsCustomEvalConfigUser: A Datadog user associated with a custom evaluator configuration.
        - `email` string: Email address of the user.
      - `llm_judge_config` LLMObsCustomEvalConfigLLMJudgeConfig: LLM judge configuration for a custom evaluator.
        - `assessment_criteria` LLMObsCustomEvalConfigAssessmentCriteria: Criteria used to assess the pass/fail result of a custom evaluator.
          - `max_threshold` number, double, nullable: Maximum numeric threshold for a passing result.
          - `min_threshold` number, double, nullable: Minimum numeric threshold for a passing result.
          - `pass_values` string[], nullable: Specific output values considered as a passing result.
          - `pass_when` boolean, nullable: When true, a boolean output of true is treated as passing.
        - `context_query` string, nullable: Query used to extract additional context for the evaluation.
        - `inference_params` LLMObsCustomEvalConfigInferenceParams, required: LLM inference parameters for a custom evaluator.
          - `frequency_penalty` number, double: Frequency penalty to reduce repetition.
          - `max_tokens` integer: Maximum number of tokens to generate.
          - `presence_penalty` number, double: Presence penalty to reduce repetition.
          - `temperature` number, double: Sampling temperature for the LLM.
          - `top_k` integer: Top-k sampling parameter.
          - `top_p` number, double: Top-p (nucleus) sampling parameter.
        - `last_used_library_prompt_template_name` string, nullable: Name of the last library prompt template used.
        - `modified_library_prompt_template` boolean, nullable: Whether the library prompt template was modified.
        - `output_schema` object, nullable: JSON schema describing the expected output format of the LLM judge. Omit for a `deterministic` evaluator.
        - `parsing_type` 'structured_output' | 'json' | 'keyword_search' | 'typed_questions': Output parsing type for a custom LLM judge evaluator.
        - `prompt_template` LLMObsCustomEvalConfigPromptMessage[]: List of messages forming the LLM judge prompt template. Omit for a `deterministic` evaluator.
          - `content` string: Text content of the message.
          - `contents` LLMObsCustomEvalConfigPromptContent[]: Multi-part content blocks for the message.
            - `type` string, required: Content block type.
            - `value` LLMObsCustomEvalConfigPromptContentValue, required: Value of a prompt message content block.
              - …
          - `role` string, required: Role of the message author.
        - `target_query` string, nullable: Query used to extract the target value to evaluate.
        - `user_specified_json_post_processing_function` string, nullable: User-provided JavaScript function applied to post-process the JSON output of the LLM judge. For a `deterministic` evaluator it is the evaluator itself: it receives the evaluated span (span scope), `{spans}` (trace scope), or `{session_id, traces}` (session scope), and its return value sets the result. A boolean, number, string or object becomes a boolean, score, categorical or JSON metric; `null` skips the evaluation; and `{value, assessment, reasoning, metadata}` also sets pass or fail, reasoning, and metadata.
      - `llm_provider` LLMObsCustomEvalConfigLLMProvider: LLM provider configuration for a custom evaluator.
        - `bedrock` LLMObsCustomEvalConfigBedrockOptions: AWS Bedrock-specific options for LLM provider configuration.
          - `inference_profile` string: Bedrock inference profile identifier, such as an application inference profile ARN.
          - `region` string: AWS region for Bedrock.
        - `integration_account_id` string: Integration account identifier.
        - `integration_provider` 'openai' | 'amazon_bedrock' | 'anthropic' | 'azure_openai' | 'vertex_ai' | 'llmproxy' | 'deterministic' | 'typesafe' | 'datadog' | 'datadog_ai_gateway' | 'insights_sponsored': Name of the LLM integration provider. Use `deterministic` for a code evaluator, which runs `llm_judge_config.user_specified_json_post_processing_function` instead of calling an LLM. `insights_sponsored` is returned for evaluators that Datadog manages and cannot be set.
        - `model_name` string: Name of the LLM model. Use `javascript` for a `deterministic` evaluator.
        - `vertex_ai` LLMObsCustomEvalConfigVertexAIOptions: Google Vertex AI-specific options for LLM provider configuration.
          - `location` string: Google Cloud region.
          - `project` string: Google Cloud project ID.
      - `target` LLMObsCustomEvalConfigTarget: Target application configuration for a custom evaluator.
        - `application_name` string, required: Name of the ML application this evaluator targets.
        - `enabled` boolean, required: Whether the evaluator is active for the target application.
        - `eval_scope` 'span' | 'trace' | 'trace_fan_out' | 'session': Scope at which to evaluate spans. `trace_fan_out` requires `llm_judge_config.target_query`.
        - `experiment_project_ids` string[]: Experiment project IDs this evaluator is scoped to.
        - `filter` string, nullable: Filter expression to select which spans to evaluate.
        - `root_spans_only` boolean, nullable: When true, only root spans are evaluated.
        - `sampling_percentage` number, double, nullable: Percentage of traces to evaluate. Must be greater than 0 and at most 100.
      - `updated_at` string, date-time, required: Timestamp when the evaluator configuration was last updated.
    - `id` string, required: Unique name identifier of the evaluator configuration.
    - `type` 'evaluator_config', required: Type of the custom Agent Observability evaluator configuration resource.

## Other responses

- `401`: Unauthorized
- `403`: Forbidden
- `429`: Too many requests

---

[API](https://skmtc.dev/datadog/apis/datadog-api-v2-collection.md) - [All operations](https://skmtc.dev/datadog/apis/datadog-api-v2-collection/llms.txt) - [OpenAPI document](https://skmtc.dev/datadog/apis/datadog-api-v2-collection/revisions/3f70db624329?raw)