--- title: "Generate a simulation suite" method: POST path: "/eval/simulation/onboarding/suite" tags: ["Simulation Suites"] --- # Generate a simulation suite `POST /eval/simulation/onboarding/suite` ## Request - Base URL: `https://api.vapi.ai` - URL: `https://api.vapi.ai/eval/simulation/onboarding/suite` - Auth: HTTP bearer Generate and save a simulation suite from reviewed test requirements, and assign it to an assistant. First draft requirements with `POST /eval/simulation/onboarding/requirements`. The response includes the saved suite, its simulations, and requirement coverage, including any gaps. Requests can take up to 3 minutes. Set a client timeout of at least 200 seconds. Use a unique `idempotencyKey` for each suite generation request. Retrying with the same key does not create a duplicate suite: - On a timeout or a 5xx (502, 504, 524), retry with the same key. You get the saved suite, a 409 while it is still generating, or a fresh generation if the first one never saved. - On a 409 `SIMULATION_ONBOARDING_GENERATION_IN_PROGRESS`, the first request is still generating. Wait `Retry-After` seconds (15) and retry with the same key. - On a 409 `SIMULATION_ONBOARDING_IDEMPOTENCY_KEY_REUSED`, the key was used for a different assistant or requirements. Send a new key. Review the generated simulations and coverage gaps. Revise the suite with `PATCH /eval/simulation/onboarding/suite/{suiteId}`, then run it with `POST /eval/simulation/run`. Each organization can generate 20 suites per UTC day, shared with the dashboard, and 2 at a time. A retry that returns a saved suite does not count toward these limits. On a 429 `SIMULATION_ONBOARDING_CONCURRENCY_LIMITED`, wait `Retry-After` seconds (30) before retrying. On a 429 `SIMULATION_ONBOARDING_QUOTA_EXCEEDED`, wait until the daily limit resets at 00:00 UTC before retrying. ## Request body - SimulationOnboardingSuiteCreateDTO - `assistantId` string, uuid, required: Assistant the suite tests and is assigned to. - `idempotencyKey` string, required: A unique key for this suite generation request. Retrying with the same key returns the saved suite, or a 409 while generation is in progress. Use a new key for a different assistant or requirements. - `requirements` SimulationOnboardingRequirementDTO[], required: The reviewed test requirements to generate simulations for, from the requirements draft. Each requirement must have a unique id. - `id` string, required: Stable requirement id, e.g. `R3`. - `text` string, required: One observable behaviour the assistant must get right. - `tag` 'assumption' | 'not-testable' | 'voice-only': Indicates an assumption, a requirement that needs a voice simulation, or a requirement that cannot be tested. - `note` string: What was assumed, why only a voice run can check it, or why it cannot be tested. ## Response `200` - SimulationOnboardingSuiteViewDTO - `suite` SimulationOnboardingSuiteRefDTO, required - `id` string, required: Suite id; pass it to edit or run the suite. - `name` string, required: Suite name. - `assistantId` string, required: Assistant the suite is assigned to. - `simulations` SimulationOnboardingSuiteSimulationDTO[], required: The suite's simulations, in order. - `id` string, required: Simulation id; pass it back to edit or remove the simulation. - `scenarioId` string, required: Scenario the simulation runs. - `name` string, required: Scenario name. - `personalityName` string: Name of the simulated caller's personality. - `successCriteria` string[], required: Each success criterion, as the yes/no question it is judged by. - `requirementIds` string[], required: Requirement ids this simulation covers, e.g. `R3`. - `mockedTools` string[], required: Tools the scenario mocks. - `coverage` SimulationOnboardingSuiteCoverageDTO, required - `covered` number, required: Requirements at least one simulation covers. - `total` number, required: Requirements in the agreed draft. - `gaps` SimulationOnboardingCoverageGapDTO[], required: Requirements no simulation covers, and why. - `requirementId` string, required: Requirement the gap is for, e.g. `R4`. - `reason` 'not-testable' | 'unmapped' | 'failed' | 'budget', required: Why no simulation covers it. `failed` and `budget` gaps are regenerated by an edit with `retryFailed`. - `detail` string: A short explanation for the user. - `retryable` number, required: Planned scenarios an edit with `retryFailed` would regenerate. - `requirements` SimulationOnboardingRequirementDTO[], required: The test requirements used to generate the suite. Match their ids to `coverage.gaps` to identify uncovered requirements. - `id` string, required: Stable requirement id, e.g. `R3`. - `text` string, required: One observable behaviour the assistant must get right. - `tag` 'assumption' | 'not-testable' | 'voice-only': Indicates an assumption, a requirement that needs a voice simulation, or a requirement that cannot be tested. - `note` string: What was assumed, why only a voice run can check it, or why it cannot be tested. - `docs` SimulationOnboardingSuiteDocsDTO, required - `bestPractices` string, required: Link to testing best practices. - `toolMocks` string, required: Link explaining why tools are mocked by default. - `reused` boolean, required: True when the request matched one already applied (same `idempotencyKey` or `patchKey`) and nothing changed. - `failedEditSimulationIds` string[]: Simulations an edit asked to rewrite that were not rewritten, for example because the rewrite did not generate in time. Retry them with a new `patchKey`. - `toolGaps` SimulationOnboardingToolGapDTO[]: Tool paths the suite's simulations do not exercise as configured, such as a tool mocked although it runs natively. Omitted when there are none. - `toolName` string, required: Function name of the tool, e.g. `handoff_to_group_1` or `transferCall`. - `assistantName` string: Assistant the tool belongs to, when it is a handoff destination's tool. Unset for the assistant's own tools. - `reason` 'dynamic-handoff' | 'destination-not-plain' | 'destination-unresolved' | 'destination-truncated' | 'transfer-external' | 'destination-name-unscoped' | 'native-tool-shadowed', required: Why the path is not exercised: a reason the simulations mock the tool, or why a native path is never reached. New reasons may be added. - `destinationNames` string[]: Static destinations the simulations do not reach through this tool. ## Other responses - `400`: The body is invalid, or the generated suite uses something the org's data-region policy doesn't allow, so it isn't saved. - `403`: The key can't create simulation suites. - `404`: The assistant wasn't found, or the org doesn't have API access to these endpoints yet. - `409`: `SIMULATION_ONBOARDING_GENERATION_IN_PROGRESS`: a request with this key is still generating; retry with the same key after `Retry-After`. `SIMULATION_ONBOARDING_IDEMPOTENCY_KEY_REUSED`: the key was used for other requirements; send a new key. - `422`: None of the requirements can be checked in a chat simulation, or an MCP server's tools could not be listed, so simulations would call them for real. Fix the requirements or the server and retry. - `429`: `SIMULATION_ONBOARDING_CONCURRENCY_LIMITED`: the org already has 2 suites generating; retry after `Retry-After`. `SIMULATION_ONBOARDING_QUOTA_EXCEEDED`: the org generated 20 suites today; don't retry, the limit resets at 00:00 UTC. - `502`: The model failed to generate a usable result. Retry. - `503`: Simulation generation is paused, no caller personalities are configured, or the quota store is unavailable for a private-key request. Quota outages return SIMULATION_ONBOARDING_QUOTA_UNAVAILABLE with Retry-After; retry with the same key. - `504`: Generation took too long. Retry with the same `idempotencyKey`: it may still save. ## Changes > 164 revisions in range; 15 not diffed, 8 could not be searched. - **2026-10-29** `d4ae2ac5e134`: 2 info - api tag `Simulation Suites` added - api tag `Simulations` removed [Operation history](https://skmtc.dev/vapiai/apis/vapi-api/changes/eval/simulation/onboarding/suite/post.md) --- [API](https://skmtc.dev/vapiai/apis/vapi-api.md) - [All operations](https://skmtc.dev/vapiai/apis/vapi-api/llms.txt) - [OpenAPI document](https://skmtc.dev/vapiai/apis/vapi-api/revisions/d73a715574fb?raw)