TRtraceloop
evaluators

Create a custom LLM-as-a-judge evaluator

Changed on

Create a new custom evaluator at the organization level without project or environment bindings. The evaluator uses LLM-as-a-judge to evaluate inputs based on the provided configuration. The response format is automatically generated from the output schema.

post/v2/evaluators

Request

Request body

descriptionstring
frequency_penaltynumber
logprobsboolean
max_tokensinteger
modestring
modelstring required

LLM parameters (from BaseLLMConfig)

namestring required

Basic fields

presence_penaltynumber
providerstring required

Provider configuration

slugstring
stopstring[]
temperaturenumber
top_pnumber

Response

OK

bindingsstring[]
evaluator_idstring
slugstring
type'AGENT_GOAL_ACCURACY' | 'CHAR_COUNT' | 'CHAR_COUNT_RATIO' | 'ANSWER_COMPLETENESS' | 'FAITHFULNESS' | 'JSON_VALIDATOR' | 'PERPLEXITY' | 'PII_DETECTOR' | 'PLACEHOLDER_REGEX' | 'PROFANITY_DETECTOR' | 'SEXISM_DETECTOR' | 'PROMPT_INJECTION' | 'TONE_DETECTION' | 'PROMPT_PERPLEXITY' | 'TOXICITY_DETECTOR' | 'ANSWER_RELEVANCY' | 'ANSWER_CORRECTNESS' | 'REGEX_VALIDATOR' | 'SECRETS_DETECTOR' | 'SQL_VALIDATOR' | 'SEMANTIC_SIMILARITY' | 'TOPIC_ADHERENCE' | 'UNCERTAINTY_DETECTOR' | 'WORD_COUNT' | 'WORD_COUNT_RATIO' | 'INSTRUCTION_ADHERENCE' | 'AGENT_FLOW_QUALITY' | 'AGENT_EFFICIENCY' | 'AGENT_GOAL_COMPLETENESS' | 'CONVERSATION_QUALITY' | 'INTENT_CHANGE' | 'CONTEXT_RELEVANCE' | 'AGENT_TOOL_ERROR_DETECTOR' | 'AGENT_TOOL_TRAJECTORY' | 'HTML_COMPARISON' | 'LLM_AS_A_JUDGE'
version_idstring

Changes