--- title: "Diagnose Eval Results" method: POST path: "/v1/api-performance/diagnose" tags: ["Eval Results"] --- # Diagnose Eval Results `POST /v1/api-performance/diagnose` ## Request - Base URL: `https://api.lightsage.com` - URL: `https://api.lightsage.com/v1/api-performance/diagnose` - Auth: API key in header `X-Lightsage-Api-Key` Analyzes recorded evaluation results in the requested window and returns ranked failure patterns. This endpoint does not start new evaluations. ## Request body - DiagnoseEvalRunsRequest - `since` string: The lookback window, such as `20d`, `72h`, or an ISO 8601 timestamp. - `format` 'json' | 'markdown' | 'md' - `include_examples` boolean - `max_examples` integer ## Response `200` Successful Response - DiagnoseEvalRunsResponse - `company` PublicCompany, required - `name` string, required - `domain` string, nullable - `logo_url` string, nullable - `relationship` string, nullable: The company relationship: `primary` for the organization's own company or `competitor` for a tracked competitor. - `since` string, required - `summary` DiagnosisSummary, required - `total_runs` integer - `failed_runs` integer - `passed_runs` integer - `failure_rate` number, nullable - `top_failed_endpoints` object[] - `top_failed_models` object[] - `top_failure_reasons` object[] - `failure_analysis` FailureAnalysisSnapshot, required - `total_runs` integer - `failed_runs` integer - `failure_rate` number, nullable - `patterns` FailurePatternSnapshot[] - `pattern` string, required - `label` string, required - `description` string, nullable - `count` integer, required - `percentage` number, nullable - `severity` string, nullable - `example_reasons` string[] - `affected_endpoints` object[] - `affected_models` string[] - `run_ids` string[]: Result IDs, usable with `GET /v1/api-performance/runs/{run_id}`. - `top_failure_reasons` object[] - `endpoints_by_failure_count` object[] - `examples` EvalRunDetail[]: Example failed results, included only when `include_examples` is true. - `id` string, required: Result ID, not an eval job ID. - `endpoint` union - PublicEndpoint - `id` string, required: Same value as `operation_id` on eval configurations. - `method` string, nullable - `path` string, nullable - `description` string, nullable - `category` object, nullable: The category object containing `id`, `name`, `slug`, and `sort_order`; null on list rows. - object - `source` EvalRunSource - `id` string, required - `type` string, nullable: The source type: `rest_api`, `python_sdk`, `node_sdk`, or `cli`. - `label` string, nullable - `metadata` object: Public source metadata such as `api_base_url` and `docs_url`; secret values are never included. - `eval_type` string, nullable: The evaluation type, such as `integration`. - `model_id` string, nullable: Public model ID, such as `openai/gpt-5.4` or `anthropic/claude-sonnet-5`. - `model_type` string, nullable: The model type: `agent` or `text`. - `agent_id` string, nullable: Coding-agent harness, such as `pi`, `kilo`, `codex`, or `claude-code`; null for text-model runs. - `passed` boolean, nullable - `score` number, nullable: A score from 0 to 1. Use `passed`, not this score, for the verdict. - `failure_reasons` string[], nullable: Labels of scoring checks that did not pass, such as `exit_code_zero` or `task_completed`. Populated whenever a check failed, including on results that passed overall, and null when every check passed. Deductions from checks that passed, such as a judge scoring 0.98, are not listed here; read `grader_results[].deductions` on the retrieved result. - `latency_ms` integer, nullable - `run_at` string, nullable - `environment` PublicEvalRunEnvironment: Non-secret summary of the environment an eval run was queued with. - `id` string, nullable: Environment ID, as listed by the environments operation. - `name` string, nullable: Environment name when the run was queued. - `updated_at` string, nullable: Environment version the run used: when its name or variables last changed before the run was queued. Null if the environment had not changed since versioning was introduced. - `queued_at` string, nullable: When the run was queued with this environment. - `variable_names` string[]: Variable names supplied to the run. Values are never returned. - `inline` boolean: True when variables were supplied inline with the run request instead of from a saved environment. - `changed_since_queued` boolean, nullable: True when the saved environment has been edited or deleted since the run was queued, so a rerun would use different values. Null when unknown or for inline environments. - `prompt_sent` string, nullable - `response_text` string, nullable - `execution_stdout` string, nullable: Output of executing the agent's code. - `execution_stderr` string, nullable: Errors from executing the agent's code. - `exit_code` integer, nullable: The process exit code, or null for evals that do not execute code. - `tool_calls` object[], nullable: One entry per tool invocation, in call order. Each carries `tool_name`, `tool_id`, `input`, and `output`; a call and its result are merged into one entry, so the list length is the invocation count the app shows as tool calls. - `grader_results` PublicGraderResult[]: Per-check verdicts behind `score`, including deductions recorded by passing checks. Not populated in diagnosis examples; retrieve the result. - `name` string, required: Grader name as configured on the eval. - `label` string, nullable: Display name for the check, when the grader sets one. - `passed` boolean, nullable: Whether this check passed. - `score` number, nullable: Score from 0 to 1 for checks that score partially, such as judges; null for pass/fail checks. - `deductions` string[]: Specific shortcomings the check recorded, even when it passed. A judge can pass with a score below 1 and still list deductions here; those checks do not appear in `failure_reasons`. - `details` string, nullable: Reasoning or check output explaining the verdict. - `criteria_verdicts` PublicCriterionVerdict[]: Judges only: a verdict for each mandatory requirement in the judge criteria. Empty for other checks and for runs graded before per-requirement verdicts were recorded. - `criterion` string, required: One mandatory requirement from the judge criteria. - `verdict` 'PASS' | 'FAIL' | 'NOT_VERIFIED' | 'NOT_APPLICABLE', required: `PASS` when the evidence shows the requirement was met, `FAIL` when it shows it was not, and `NOT_VERIFIED` when the evidence cannot confirm it. Any `FAIL` or `NOT_VERIFIED` fails the run, whatever the score. `NOT_APPLICABLE` when the requirement states its own condition ("If X is used, ...") and that condition did not occur; it neither passes nor fails the run. - `evidence` string, nullable: Short evidence the judge gave for the verdict. - `not_applicable` boolean: Judges only: true when every requirement was `NOT_APPLICABLE`. The check is left out of the run's score and does not fail the run. - `raw` object, nullable - `markdown` string, nullable: The Markdown report, or null when `format` is `json`. ## Other responses - `422`: Validation Error ## Changes > 8 revisions in range; 1 not diffed. - **2026-11-06** `14890dfe867e`: 1 warning, 1 info - added the new `NOT_APPLICABLE` enum value to the `examples/items/grader_results/items/criteria_verdicts/items/verdict` response property for the response status `200` - added the optional property `examples/items/grader_results/items/not_applicable` to the response with the `200` status [Operation history](https://skmtc.dev/usesapient/apis/sapient-public-api/changes/v1/api-performance/diagnose/post.md) --- [API](https://skmtc.dev/usesapient/apis/sapient-public-api.md) - [All operations](https://skmtc.dev/usesapient/apis/sapient-public-api/llms.txt) - [OpenAPI document](https://skmtc.dev/usesapient/apis/sapient-public-api/revisions/14890dfe867e?raw)