BYO-LLM

Evaluate an LLM provider's SUITABILITY for legal work (owner-only) · 0 credits

Beyond the connectivity health check (/test): it runs the configured engine against an ADVERSARIAL legal mini-goldset (ES∪EN) and scores it with Nexus's own verification engine. It measures citation anti-fabrication, sense fidelity, instruction following and reliability. Returns { rating, score, dimensions, reliability, recommendation, probes, assessed?, assessedCode?, assessedNote?, scoreCoverage?, unmeasured? }. Owner-only, 0 credits; it re-validates SSRF; errors are sanitized (never an echo of the credential). Configure the provider first (POST /api/v1/llm-providers) and evaluate its {id}. A warn/fail judges the MODEL chosen, not the platform: citation verification runs over the output whatever the engine is.

⚠️ Check assessed before reading rating or score. The scorers detect BAD behaviour (a fabricated citation, an inverted meaning, a broken format), so an engine that returns nothing gives them nothing to catch. A probe with no usable output is NOT scored (probes[].evaluable: false), it leaves its dimension's denominator, and the weight that could not be measured is never credited: score runs out of 100 x scoreCoverage. assessed: false + assessedCode say WHY (engine did not answer / answered empty / partial coverage). An engine that does not answer does not score low — it does not score. Scope: mcp | * (NOT cases).

post/api/v1/llm-providers/{id}/suitability

Request

  • Base URL: https://legal.nexusquantum.legal
  • URL: https://legal.nexusquantum.legal/api/v1/llm-providers/{id}/suitability
  • Auth: HTTP bearer

Path parameters

idstring uuid required

Response

Suitability report (rating pass|warn|fail + dimensions + recommendation).

rating'pass' | 'warn' | 'fail'

Overall suitability verdict. READ assessed FIRST: with assessed: false this verdict is the non-certifying value of a CLOSED enum, not a measurement of your model.

scoreinteger

Score 0-100 (weighted sum of the dimensions) OVER THE MEASURED PART: the real ceiling is 100 x scoreCoverage, not 100. Whatever could not be measured adds 0 — it is never credited as a pass. score: 0 with scoreCoverage: 0 means "nothing was measured", NOT "measured and scored zero".

modelstring

Model evaluated.

dimensionsobject

Per dimension { pass, total, ok, unusable? }: anti_fabrication, faithfulness, instruction_following, reliability. total is the MEASURED denominator (probes that produced usable output); unusable counts the probes discarded for lack of it, so total + unusable = probes launched. ok: true requires the dimension to have been measured AND fully passed — a dimension with total: 0 is ok: false, since absence of evidence never certifies.

reliabilityobject

{ calls, errors, errorRate, p95LatencyMs, unusable? } over the goldset calls (captures the down/unstable-engine failure class). errors = transport failures; unusable = calls that yielded no usable output (a SUPERSET of errors: it also counts 200s answered with empty content).

recommendationstring

Plain-language verdict + recommended action. When no quality dimension could be measured it carries the English assessedNote instead of a verdict on the model.

probesobject[]

Per-probe detail (id, dimension, lang, ok, pass, note, evaluable?). evaluable: false = the engine produced no usable output for that probe, so it was NOT scored: the accompanying pass: false is filler, it means "not measured", not "measured and failed".

assessedboolean

ADDITIVE and optional. false = the report is NOT a complete measurement: at least one probe (or a whole dimension) could not be scored because your engine returned no usable output. An engine that answers nothing does not score low — IT DOES NOT SCORE: without this field a mute engine collected the full 40% of anti_fabrication ("it fabricated no citation" — it wrote no word at all) and the 15% of reliability, and came out rated as usable. It is a new optional FIELD on purpose, never a new value inside the closed rating enum, so integrators that switch on rating do not break.

assessedCode'ENGINE_NO_RESPONSE' | 'ENGINE_EMPTY_OUTPUT' | 'PARTIAL_COVERAGE'

Stable machine code, present only when assessed: false. ENGINE_NO_RESPONSE = your engine did not answer (credentials, balance/quota, reachability). ENGINE_EMPTY_OUTPUT = it answered with no content (model id, output token budget, content filter). PARTIAL_COVERAGE = some probes were not scored.

assessedNotestring

Human-readable explanation (English, prefixed with assessedCode), present only when assessed: false.

scoreCoveragenumber

0..1 — share of the total weight that COULD be measured. score is out of 100 x scoreCoverage. A value below 1 means part of the report is a gap, not a result.

unmeasuredstring[]

Dimensions with NO evaluable probe: they were not measured at all, neither well nor badly. Absent when everything was measured.

Changes

No recorded changes to this operation across all 1 revision of this API.