Simulation Suites

Generate a simulation suite

Changed on

Generate and save a simulation suite from reviewed test requirements, and assign it to an assistant. First draft requirements with POST /eval/simulation/onboarding/requirements. The response includes the saved suite, its simulations, and requirement coverage, including any gaps.

Requests can take up to 3 minutes. Set a client timeout of at least 200 seconds.

Use a unique idempotencyKey for each suite generation request. Retrying with the same key does not create a duplicate suite:

  • On a timeout or a 5xx (502, 504, 524), retry with the same key. You get the saved suite, a 409 while it is still generating, or a fresh generation if the first one never saved.
  • On a 409 SIMULATION_ONBOARDING_GENERATION_IN_PROGRESS, the first request is still generating. Wait Retry-After seconds (15) and retry with the same key.
  • On a 409 SIMULATION_ONBOARDING_IDEMPOTENCY_KEY_REUSED, the key was used for a different assistant or requirements. Send a new key.

Review the generated simulations and coverage gaps. Revise the suite with PATCH /eval/simulation/onboarding/suite/{suiteId}, then run it with POST /eval/simulation/run.

Each organization can generate 20 suites per UTC day, shared with the dashboard, and 2 at a time. A retry that returns a saved suite does not count toward these limits. On a 429 SIMULATION_ONBOARDING_CONCURRENCY_LIMITED, wait Retry-After seconds (30) before retrying. On a 429 SIMULATION_ONBOARDING_QUOTA_EXCEEDED, wait until the daily limit resets at 00:00 UTC before retrying.

post/eval/simulation/onboarding/suite

Request

  • Base URL
    https://api.vapi.ai
  • Auth
    HTTP bearer

Request body

assistantIdstring uuid required

Assistant the suite tests and is assigned to.

idempotencyKeystring required

A unique key for this suite generation request. Retrying with the same key returns the saved suite, or a 409 while generation is in progress. Use a new key for a different assistant or requirements.

Response

assistantIdstring required

Assistant the suite is assigned to.

reusedboolean required

True when the request matched one already applied (same idempotencyKey or patchKey) and nothing changed.

failedEditSimulationIdsstring[]

Simulations an edit asked to rewrite that were not rewritten, for example because the rewrite did not generate in time. Retry them with a new patchKey.

Changes

Some revisions have not been checked yet.