---
title: "Start a proving run (returns immediately, processes in background)"
method: POST
path: "/v3/proving/run"
tags: ["v3", "proving"]
---

# Start a proving run (returns immediately, processes in background)

`POST /v3/proving/run`

## Request

- Base URL: `https://api.mentium.io` (the document declares no server; this is the origin it was published from)
- URL: `https://api.mentium.io/v3/proving/run`
- Auth: HTTP bearer

## Headers

- `mentium-workspace-id` string, nullable

## Request body

- ProvingRunTrigger
  - `agent_id` string, uuid, required: Agent ID (for tracking)
  - `scenario_id` string, uuid, required: Scenario to run
  - `system_prompt_override` string, nullable: If provided, use this prompt instead of the agent's saved system prompt for the run
  - `user_prompt_override` string, nullable: If provided, use this prompt instead of the agent's saved user prompt for the run
  - `response_format_override` object, nullable: If provided, use this output schema instead of the agent's saved response format for the run
  - `tools_override` string[], nullable: If provided, use these tools instead of the agent's saved tools for the run
  - `release` string, nullable: The release a staging deploy's run tests
  - `image_tag` string, nullable: The image tag a staging deploy's run tests

## Response `202`

Successful Response

- ProvingRunResponse
  - `id` string, required
  - `agent_id` string, required
  - `agent_revision_id` string, nullable: The revision this run evaluates, when it is one of a revision's runs
  - `scenario_id` string, required
  - `agent_snapshot` ProvingAgentSnapshot, required
    - `name` string, required
    - `agent_type` string, nullable
    - `model` string, required
    - `temperature` number, nullable
    - `max_tokens` integer, nullable
    - `reasoning_effort` string, nullable
    - `system_prompt_template` string, required
    - `user_prompt_template` string, nullable
    - `tools` string[]
    - `type_config` object, nullable
    - `tool_inputs` object
    - `mcp_bindings` object[]
    - `automation_tools` object[]: Each bound webhook tool as the model is told about it; its URL, headers and secret are never stored
    - `workspace_has_email_signature` boolean
    - `tested_draft` boolean: The run tested the agent's unpublished draft, not the live agent
    - `version_number` integer, nullable: The version the run tested: the live one, the one a draft is based on, or the revision it evaluated
    - `system_prompt_template_overridden` boolean
    - `user_prompt_template_overridden` boolean
    - `tools_overridden` boolean
    - `type_config_overridden` boolean
  - `scenario_snapshot` ProvingScenarioSnapshot, required
    - `is_system` boolean
    - `severity` 'p0_safety' | 'p1_correctness' | 'p2_quality'
    - `name` string, required
    - `description` string, nullable
    - `node_type` 'conversation-agent' | 'ai-agent'
    - `node_config` object
    - `prefix_transcript_turns` ProvingTranscriptTurn[]
      - `type` 'transcript_turn'
      - `transcript_turn_idx` integer: 0-indexed index of ProvingScenario.prefix_transcript_turns and ProvingRun.transcript_turns. 0 for introductory agent turn, 1 for first user turn, 2 for second agent turn, etc. Used by backend and frontend for ordering turns.
      - `role` 'user' | 'agent', required
      - `transcript_turn_items` union[]
        - union
          - ProvingTranscriptTurnItemMessage
            - `type` 'transcript_turn_item_message'
            - `transcript_turn_item_idx` integer: 0-indexed index of ProvingTranscriptTurn.transcript_turn_items. Used by backend and frontend for ordering turn items.
            - `message_content` string, required
          - ProvingTranscriptTurnItemToolCall
            - `type` 'transcript_turn_item_tool_call'
            - `transcript_turn_item_idx` integer: 0-indexed index of ProvingTranscriptTurn.transcript_turn_items. Used by backend and frontend for ordering turn items.
            - `tool_call_id` string, required
            - `tool_call_name` string, required
            - `tool_call_params` object, required
            - `tool_call_result` object, required
    - `mock_tool_calls` ProvingMockToolCallOutput[]
      - `type` 'mock_tool_call'
      - `mock_tool_call_idx` integer: 0-indexed index of ProvingScenario.mock_tool_calls. 0 for first mock tool call, 1 for second mock tool call, etc. Used by backend and frontend for ordering tool calls. Tool calls can be repeated with the same name and parameters and can only be consumed once so needed to be ordered for priority. E.g. first tool call to delete a database record must give success result but second tool call to delete the same record must give failure result.
      - `transcript_turn_idx` integer, nullable: 0-indexed index of ProvingScenario.prefix_transcript_turns. None for a transcript-level, 0 for agent introductory turn, 1 for first user turn, 2 for second agent turn, etc.
      - `tool_call_name` string, required
      - `tool_call_params` ProvingMockToolCallParamItem[]
        - `type` 'mock_tool_call_param_item'
        - `mock_tool_call_param_item_idx` integer: 0-indexed index of ProvingMockToolCall.tool_call_params. Not used by backend since tool call parameters are functionally unordered. Used by frontend for ordering tool call parameters in the UI for consistency.
        - `tool_call_param_path` union[], required
          - union
            - string
            - integer
        - `tool_call_param_value` union, required
          - string
          - integer
          - number
          - boolean
        - `match_method` 'exact_match' | 'one_of' | 'contains' | 'in_range', required
        - `case_sensitive` boolean
      - `tool_call_result` object, required
    - `evaluations` ProvingEvaluationOutput[]
      - `type` 'evaluation'
      - `evaluation_idx` integer: 0-indexed index of ProvingScenario.evaluations. Used by backend and frontend for ordering evaluations. Evaluations are ran sequentially. E.g. agent must call google search tool with terms 'weather in San Francisco' (evaluation item 0) and 'weather in New York' (evaluation item 1) and then must respond about temerature and rain chances (evaluation item 2).
      - `name` string, required
      - `transcript_turn_idx` integer, nullable: 0-indexed index of ProvingScenario.prefix_transcript_turns. Used by backend and frontend for ordering evaluations. None for a transcript-level and must run for all turns serialized and passed to the LLM at once, 0 for agent introductory turn, 1 for first user turn, 2 for second agent turn, etc. Transcript-level evaluation limited to only LLM evaluation since message and tool call evaluation at transcript level does not make sense.
      - `operator` 'not' | 'and' | 'or', required
      - `evaluation_items` union[]
        - union
          - ProvingEvaluationItemMessage
            - `type` 'evaluation_item_message'
            - `evaluation_item_idx` integer: 0-indexed index of ProvingEvaluation.evaluation_items. Not used by backend since evaluation items are unordered. Used by frontend for ordering evaluation items in the UI. Evaluation item must run for each of the turn items and pass for one of them to pass the evaluation.
            - `name` string, required
            - `expected_message_content` string, required
            - `match_method` 'exact_match' | 'one_of' | 'contains', required
            - `case_sensitive` boolean
          - ProvingEvaluationItemToolCallOutput
            - `type` 'evaluation_item_tool_call'
            - `evaluation_item_idx` integer: 0-indexed index of ProvingEvaluation.evaluation_items. Not used by backend since evaluation items are unordered. Used by frontend for ordering evaluation items in the UI. Evaluation item must run for each of the turn items and pass for one of them to pass the evaluation.
            - `name` string, required
            - `tool_call_name` string, required
            - `expected_tool_call_params` ProvingEvaluationItemToolCallParamItem[]
              - …
          - ProvingEvaluationItemLLM
            - `type` 'evaluation_item_llm'
            - `evaluation_item_idx` integer: 0-indexed index of ProvingEvaluation.evaluation_items. Not used by backend since evaluation items are unordered. Used by frontend for ordering evaluation items in the UI. Evaluation item must run for all turn items serialized and passed to the LLM at once and pass for one of them to pass the evaluation.
            - `name` string, required
            - `expected_turn_observation` string, required: Natural language expected observation in the turn
            - `expected_turn_observation_criteria` string[]: Natural-language criteria, each judged met or not; empty judges expected_turn_observation alone.
            - `criteria_pass_fraction` number: Fraction of the criteria that must be met. Ignored when there are no criteria.
          - ProvingEvaluationItemSafetyInvariant
            - `type` 'evaluation_item_safety_invariant'
            - `evaluation_item_idx` integer: 0-indexed index of ProvingEvaluation.evaluation_items. Not used by backend since evaluation items are unordered. Used by frontend for ordering evaluation items in the UI. Evaluation item runs once over the whole conversation.
            - `name` string, required
            - `invariant_id` 'quotes_only_known_figures' | 'names_only_allowed_figures' | 'never_names_figures', required
            - `invariant_params` object: What the scenario tells the invariant, such as figures it may speak
    - `definition_digest` string, nullable: A fingerprint of the test when the run started; runs with the same one ran the same test
    - `definition_version` integer, nullable: The test version when the run started
  - `transcript_turns` ProvingTranscriptTurn[], nullable
    - `type` 'transcript_turn'
    - `transcript_turn_idx` integer: 0-indexed index of ProvingScenario.prefix_transcript_turns and ProvingRun.transcript_turns. 0 for introductory agent turn, 1 for first user turn, 2 for second agent turn, etc. Used by backend and frontend for ordering turns.
    - `role` 'user' | 'agent', required
    - `transcript_turn_items` union[]
      - union
        - ProvingTranscriptTurnItemMessage
          - `type` 'transcript_turn_item_message'
          - `transcript_turn_item_idx` integer: 0-indexed index of ProvingTranscriptTurn.transcript_turn_items. Used by backend and frontend for ordering turn items.
          - `message_content` string, required
        - ProvingTranscriptTurnItemToolCall
          - `type` 'transcript_turn_item_tool_call'
          - `transcript_turn_item_idx` integer: 0-indexed index of ProvingTranscriptTurn.transcript_turn_items. Used by backend and frontend for ordering turn items.
          - `tool_call_id` string, required
          - `tool_call_name` string, required
          - `tool_call_params` object, required
          - `tool_call_result` object, required
  - `evaluation_results` ProvingEvaluationResult[], nullable
    - `type` 'evaluation_result'
    - `evaluation_result_idx` integer: 0-indexed index of ProvingScenario.evaluation_results. Used by backend and frontend for ordering evaluation results. Evaluation results are ran sequentially.
    - `evaluation_idx` integer: 0-indexed index of ProvingScenario.evaluations. Used by backend and frontend for ordering evaluation results.
    - `transcript_turn_idx` integer, nullable: 0-indexed index of ProvingScenario.prefix_transcript_turns. Used by backend and frontend for ordering evaluation results. None for a transcript-level and must run for all turns serialized and passed to the LLM at once, 0 for agent introductory turn, 1 for first user turn, 2 for second agent turn, etc. Transcript-level evaluation limited to only LLM evaluation since message and tool call evaluation at transcript level does not make sense.
    - `evaluation_result_items` union[]
      - union
        - ProvingEvaluationResultItemMessage
          - `type` 'evaluation_result_item_message'
          - `evaluation_result_item_idx` integer: 0-indexed index of ProvingEvaluationResult.evaluation_result_items
          - `evaluation_item_idx` integer: 0-indexed index of ProvingEvaluation.evaluation_items. Not used by backend since evaluation items are unordered. Used by frontend for ordering evaluation items in the UI. Evaluation item must run for each of the turn items and pass for one of them to pass the evaluation.
          - `actual_message_content` string, nullable
          - `passed` boolean, required
          - `result_message` string, nullable
          - `error_message` string, nullable
        - ProvingEvaluationResultItemToolCall
          - `type` 'evaluation_result_item_tool_call'
          - `evaluation_result_item_idx` integer: 0-indexed index of ProvingEvaluationResult.evaluation_result_items
          - `evaluation_item_idx` integer: 0-indexed index of ProvingEvaluation.evaluation_items. Not used by backend since evaluation items are unordered. Used by frontend for ordering evaluation items in the UI. Evaluation item must run for each of the turn items and pass for one of them to pass the evaluation.
          - `actual_tool_call_params` ProvingTranscriptTurnItemToolCall[], nullable
            - `type` 'transcript_turn_item_tool_call'
            - `transcript_turn_item_idx` integer: 0-indexed index of ProvingTranscriptTurn.transcript_turn_items. Used by backend and frontend for ordering turn items.
            - `tool_call_id` string, required
            - `tool_call_name` string, required
            - `tool_call_params` object, required
            - `tool_call_result` object, required
          - `passed` boolean, required
          - `result_message` string, nullable
          - `error_message` string, nullable
        - ProvingEvaluationResultItemLLM
          - `type` 'evaluation_result_item_llm'
          - `evaluation_result_item_idx` integer: 0-indexed index of ProvingEvaluationResult.evaluation_result_items
          - `evaluation_item_idx` integer: 0-indexed index of ProvingEvaluation.evaluation_items. Not used by backend since evaluation items are unordered. Used by frontend for ordering evaluation items in the UI. Evaluation item must run for each of the turn items and pass for one of them to pass the evaluation.
          - `passed` boolean, required
          - `result_message` string, nullable
          - `error_message` string, nullable
        - ProvingEvaluationResultItemSafetyInvariant
          - `type` 'evaluation_result_item_safety_invariant'
          - `evaluation_result_item_idx` integer: 0-indexed index of ProvingEvaluationResult.evaluation_result_items
          - `evaluation_item_idx` integer: 0-indexed index of ProvingEvaluation.evaluation_items. Not used by backend since evaluation items are unordered. Used by frontend for ordering evaluation items in the UI. Evaluation item runs once over the whole conversation.
          - `violations` string[]: Each way the conversation broke the invariant
          - `passed` boolean, required
          - `result_message` string, nullable
          - `error_message` string, nullable
    - `passed` boolean, required
  - `status` 'pending' | 'running' | 'completed' | 'failed', required
  - `passed` boolean, nullable
  - `error_message` string, nullable
  - `duration_seconds` number, nullable
  - `recording_url` string, nullable: A phone run's call audio, agent left and carrier right: a one-hour link on GET /runs/{id}, null in lists and the stream
  - `release` string, nullable: The release a staging deploy's run tested
  - `image_tag` string, nullable: The image tag a staging deploy's run tested
  - `mentium_created_at` string, date-time, required
  - `mentium_updated_at` string, date-time, required

## Other responses

- `422`: Validation Error

---

[API](https://skmtc.dev/mentium/apis/mentium-api.md) - [All operations](https://skmtc.dev/mentium/apis/mentium-api/llms.txt) - [OpenAPI document](https://skmtc.dev/mentium/apis/mentium-api/revisions/b4ed8e59432e?raw)