evaluations

Run Evaluation

Run evaluation with pluggable evaluators (field_accuracy, fuzzy_field_match, llm_judge, equals_expected). Set infer_corrections=true for notes-only feedback. Returns immediately with status=running. Poll by ID for completion.

post/v1/evaluations/run

Request

  • Base URL: https://api.vlm.run (the document declares no server; this is the origin it was published from)
  • URL: https://api.vlm.run/v1/evaluations/run
  • Auth: HTTP bearer

Request body

source_typestring required

'agent', 'request_domain', or 'skill'

source_idstring required

Agent ID, domain name, or skill ID

source_labelstring required

Display label for the source

modelstring nullable

Model used to produce the evaluated responses (e.g. 'vlmrun-orion-1:fast'). Stored with the run and reported in the performance section.

execution_idsstring[] nullable

Agent execution IDs to evaluate

request_idsstring[] nullable

Request IDs to evaluate

skill_idsstring[] nullable

Skill IDs to evaluate

data_fromstring nullable

ISO date filter start

data_tostring nullable

ISO date filter end

evaluatorsstring[]

Evaluators to run; each produces its own section in the response.

infer_correctionsboolean

Use LLM to produce corrected JSON from notes-only feedback.

fieldsstring[] nullable

Restrict evaluation to these field names (top-level keys or dotted leaf-path prefixes, e.g. ['name', 'address', 'items']). When omitted, all fields are evaluated.

skip_null_expectedboolean

When true (default), fields where the ground-truth (expected) value is null are excluded from evaluation. This avoids penalizing the model for extracting values that the ground-truth left empty.

Response

Successful Response

object required

Changes

No recorded changes to this operation across all 2 revisions of this API.