ORorq-ai
Evals

Invoke a Custom Evaluator

Changed on

Runs an evaluator that already exists in the workspace. Accepts either a conversation or the structured input and output fields; when both are present the conversation wins.

post/v3/evaluators/{id}/invoke

Request

  • Base URL
    https://my.orq.ai
  • URL
    https://my.orq.ai/v3/evaluators/{id}/invoke
  • Auth
    HTTP bearer

Path parameters

idstring required

Accepts a bare id, id@version, or id@environment.

Request body

modelstring

Model to grade with, as a catalog id such as "openai/gpt-4o".

Only meaningful for a hub template of type llm_eval or ragas, which has no model of its own. A stored evaluator uses the model on its own definition and ignores this.

querystring

Latest user message. Folds into context.input.user_query.

outputstring

The generated response from the model. Folds into context.output.response.

referencestring

The reference used to compare the output. Folds into context.input.expected_output.

retrievalsstring[]

Knowledge base retrievals. Folds into context.input.retrievals.

messagesobject[]

The conversation that produced the output. Folds into context.messages.

variablesobject

Template variables for evaluator prompt substitution. Folds into context.variables.

Response

OK

typestring

Discriminator for the verdict shape: "string", "number", "boolean", "string_array", "rouge_n", "bert_score", "llm_evaluator", "http_eval".

valueunknown
trace_idstring

Trace reference of the evaluator's own span. Optional so an absent reference is omitted rather than emitted as an empty string.

span_idstring
evaluator_idstring
statusstring

How the run ended, as distinct from passed: "passed", "condition_failed", "failed" or "timed_out". A string, not an enum, because the engine owns the vocabulary.

passedboolean

The guardrail's decision when the evaluator has one, the grader's own judgement otherwise. Always present, so read guardrail_config to detect a guardrail, not this.

explanationstring
categoriesstring[]

Set by the classifying graders (moderation, PII, secret detection), which report which categories tripped rather than a single verdict.

confidencenumber double

Optional so an absent score is omitted rather than reported as 0.0, which a consumer would read as maximum uncertainty.

Changes

  • 0b4e519a544fRevision changes
    • info

      added the new optional request property

    • info

      added the new optional request property

    • info

      added the non-success response with the status

  • 732aed62833aRevision changes
    • info

      removed the non-success response with the status