--- title: "Evaluate an LLM provider's SUITABILITY for legal work (owner-only) · 0 credits" method: POST path: "/api/v1/llm-providers/{id}/suitability" tags: ["BYO-LLM"] --- # Evaluate an LLM provider's SUITABILITY for legal work (owner-only) · 0 credits `POST /api/v1/llm-providers/{id}/suitability` ## Request - Base URL: `https://legal.nexusquantum.legal` - URL: `https://legal.nexusquantum.legal/api/v1/llm-providers/{id}/suitability` - Auth: HTTP bearer Beyond the connectivity health check (`/test`): it runs the configured engine against an ADVERSARIAL legal mini-goldset (ES∪EN) and scores it with Nexus's own verification engine. It measures citation anti-fabrication, sense fidelity, instruction following and reliability. Returns `{ rating, score, dimensions, reliability, recommendation, probes, assessed?, assessedCode?, assessedNote?, scoreCoverage?, unmeasured? }`. Owner-only, 0 credits; it re-validates SSRF; errors are sanitized (never an echo of the credential). Configure the provider first (`POST /api/v1/llm-providers`) and evaluate its `{id}`. A `warn`/`fail` judges the MODEL chosen, not the platform: citation verification runs over the output whatever the engine is. ⚠️ **Check `assessed` before reading `rating` or `score`.** The scorers detect BAD behaviour (a fabricated citation, an inverted meaning, a broken format), so an engine that returns nothing gives them nothing to catch. A probe with no usable output is NOT scored (`probes[].evaluable: false`), it leaves its dimension's denominator, and the weight that could not be measured is never credited: `score` runs out of `100 x scoreCoverage`. `assessed: false` + `assessedCode` say WHY (engine did not answer / answered empty / partial coverage). An engine that does not answer does not score low — it does not score. Scope: mcp | * (NOT cases). ## Path parameters - `id` string, uuid, required ## Response `200` Suitability report (rating pass|warn|fail + dimensions + recommendation). - LlmProviderSuitabilityReport: SUITABILITY report of the BYO engine for legal work (goes BEYOND the connectivity health check in `/test`). Runs the model against an ADVERSARIAL legal mini-goldset (ES∪EN) and scores it with Nexus's OWN citation verification engine (meaning vetoes + citation extraction): it judges the model with the same wall that protects the client. A weak model NEVER produces a false-verified (verification runs over the output whatever the engine); this measures the QUALITY that a bad model degrades. Citation fabrication is disqualifying (rating `fail`). - `rating` 'pass' | 'warn' | 'fail': Overall suitability verdict. READ `assessed` FIRST: with `assessed: false` this verdict is the non-certifying value of a CLOSED enum, not a measurement of your model. - `score` integer: Score 0-100 (weighted sum of the dimensions) OVER THE MEASURED PART: the real ceiling is `100 x scoreCoverage`, not 100. Whatever could not be measured adds 0 — it is never credited as a pass. `score: 0` with `scoreCoverage: 0` means "nothing was measured", NOT "measured and scored zero". - `model` string: Model evaluated. - `dimensions` object: Per dimension `{ pass, total, ok, unusable? }`: anti_fabrication, faithfulness, instruction_following, reliability. `total` is the MEASURED denominator (probes that produced usable output); `unusable` counts the probes discarded for lack of it, so `total + unusable` = probes launched. `ok: true` requires the dimension to have been measured AND fully passed — a dimension with `total: 0` is `ok: false`, since absence of evidence never certifies. - `reliability` object: `{ calls, errors, errorRate, p95LatencyMs, unusable? }` over the goldset calls (captures the down/unstable-engine failure class). `errors` = transport failures; `unusable` = calls that yielded no usable output (a SUPERSET of `errors`: it also counts 200s answered with empty content). - `recommendation` string: Plain-language verdict + recommended action. When no quality dimension could be measured it carries the English `assessedNote` instead of a verdict on the model. - `probes` object[]: Per-probe detail (id, dimension, lang, ok, pass, note, evaluable?). `evaluable: false` = the engine produced no usable output for that probe, so it was NOT scored: the accompanying `pass: false` is filler, it means "not measured", not "measured and failed". - `assessed` boolean: ADDITIVE and optional. `false` = the report is NOT a complete measurement: at least one probe (or a whole dimension) could not be scored because your engine returned no usable output. An engine that answers nothing does not score low — IT DOES NOT SCORE: without this field a mute engine collected the full 40% of `anti_fabrication` ("it fabricated no citation" — it wrote no word at all) and the 15% of `reliability`, and came out rated as usable. It is a new optional FIELD on purpose, never a new value inside the closed `rating` enum, so integrators that `switch` on rating do not break. - `assessedCode` 'ENGINE_NO_RESPONSE' | 'ENGINE_EMPTY_OUTPUT' | 'PARTIAL_COVERAGE': Stable machine code, present only when `assessed: false`. `ENGINE_NO_RESPONSE` = your engine did not answer (credentials, balance/quota, reachability). `ENGINE_EMPTY_OUTPUT` = it answered with no content (model id, output token budget, content filter). `PARTIAL_COVERAGE` = some probes were not scored. - `assessedNote` string: Human-readable explanation (English, prefixed with `assessedCode`), present only when `assessed: false`. - `scoreCoverage` number: 0..1 — share of the total weight that COULD be measured. `score` is out of `100 x scoreCoverage`. A value below 1 means part of the report is a gap, not a result. - `unmeasured` string[]: Dimensions with NO evaluable probe: they were not measured at all, neither well nor badly. Absent when everything was measured. ## Other responses - `404`: NOT_FOUND (does not exist or does not belong to your firm) - `409`: PROVIDER_NOT_RESOLVABLE (the config could not be resolved, e.g. disabled) - `422`: INVALID_PROVIDER_URL (base_url fails the SSRF filter on re-validation) - `500`: SUITABILITY_FAILED / PROVIDER_RESOLVE_FAILED --- [API](https://skmtc.dev/nexusquantum/apis/nexus-legal-api.md) · [All operations](https://skmtc.dev/nexusquantum/apis/nexus-legal-api/llms.txt) · [OpenAPI document](https://skmtc.dev/nexusquantum/apis/nexus-legal-api/revisions/4084d8d4eb0b?raw)