---
title: "Run recovery for a single failed resource"
method: POST
path: "/simulations/{simulationId}/recover-resource"
tags: ["Simulations"]
---

# Run recovery for a single failed resource

`POST /simulations/{simulationId}/recover-resource`

## Request

- Base URL: `https://www.cloudworldmodel.ai/api`
- URL: `https://www.cloudworldmodel.ai/api/simulations/{simulationId}/recover-resource`
- Auth: one of:
  - HTTP bearer
  - API key in header `X-PAYMENT`
  - API key in header `PAYMENT-SIGNATURE`

Clears a specific failed resource's failure state so the simulation
engine restores it to healthy over subsequent steps. Deactivates any
in-effect **reversible** failure injection targeting the resource
(`instance_down`, `database_overload`), restores snapshotted
characteristics where available, and resets the resource's health
counters to zero so only subsequent qualifying simulation steps count.

Returns 409 `PERMANENT_DATA_LOSS` for destroyed data that has not been
explicitly rebuilt. Generic recovery never clears `dataState`; an empty
rebuilt database can serve again while its original data remains lost.

The quick `inject-failure` endpoint is intentionally different: its
positive `failureParkSteps` window is automatic, so the node may rejoin
routing after the window without this endpoint. Use `recover-resource`
when an explicit recovery is required for a persistent reversible
`instance_down` failure; poll `isRoutable`, `routedRps`, and
`recoveryProgress` after either flow.

**`instance_kill` cannot be recovered** — that failure type
permanently removes the target resource from the simulation. Calling
this endpoint with the killed resource's id returns `400
RESOURCE_KILLED`; re-add the resource via
`PATCH /simulations/{simulationId}`, or prefer `instance_down` when
you want a reversible outage.

**Scoped to the targeted resource only** — other resources that remain
failed are not affected.

The response echoes the applied outcome, including `stepsToHealthy`:
a lower-bound estimate of the `POST /step` (or `step-hybrid`) calls
needed for the engine to demote the resource back to `healthy` under
the current recovery policy (defaults: critical→warning after 4 steps
below 80% CPU, warning→healthy after 3 steps below 70%; the default
lower bound is 7 steps from critical). `stepsToHealthyIsLowerBound` is always true:
explicitly failed resources may require an additional failure-park
transition before cooldown resumes, and a CPU threshold breach resets
the cooldown counter. Callers should take at least `stepsToHealthy`
steps, then poll `GET /simulations/{simulationId}` until the resource
is healthy.

Works for every provider (AWS, GCP, Azure, OCI, DigitalOcean) —
unlike `bulk-resize`, which is DigitalOcean-only.

Provide `resourceName` or `resourceId`. Returns 400 when the resource
cannot be found, was permanently removed by `instance_kill`, or is
already healthy.

Requires `write` scope and ownership.

## Path parameters

- `simulationId` string, required

## Request body

- object
  - `resourceName` string: Name of the failed resource to recover (case-insensitive exact match). Provide this or resourceId.
  - `resourceId` string: ID of the failed resource to recover. Provide this or resourceName.

## Response `200`

Recovery initiated for the targeted resource

- object
  - `resolvedResourceId` string
  - `resolvedResourceName` string
  - `previousHealth` string: Resource status before recovery (e.g. critical, warning)
  - `recoveryState` 'recovering'
  - `stepsToHealthy` integer: Lower-bound estimate — callers should take at least this many simulation steps before polling for healthy. Explicitly failed resources may require an additional failure-park transition before cooldown resumes, and a CPU threshold breach resets the cooldown counter.
  - `stepsToHealthyIsLowerBound` boolean: Always true; stepsToHealthy is a lower bound, not a completion guarantee.
  - `recoveryProgress` RecoveryProgress: Read-only recovery progress derived from the engine's current resource state. Poll `GET /simulations/{simulationId}` or the next step response until `state` is `healthy`; `stepsToHealthy` is only a lower-bound estimate.
    - `state` 'parked' | 'blocked' | 'cooling_down' | 'healthy', required: Current recovery stage; blocked means an active failure prevents cooldown.
    - `parkWindow` object, required: Progress through the forced-unavailability window before cooldown recovery can advance.
      - `totalSteps` integer, required: Total forced-unavailability steps for an explicitly failed compute/Kubernetes resource; zero when not applicable.
      - `completedSteps` integer, required: Park-window steps completed so far.
      - `remainingSteps` integer, required: Park-window steps remaining before cooldown can advance.
    - `cooldown` object, required: Progress through the current status-demotion cooldown window.
      - `target` 'warning' | 'healthy' | 'null', nullable, required: Status targeted by the current cooldown, or null when already healthy.
      - `completedSteps` integer, required: Consecutive qualifying simulation steps already completed toward the cooldown target.
      - `requiredSteps` integer, required: Consecutive qualifying steps required to reach target.
      - `remainingSteps` integer, required: Qualifying cooldown steps still needed.
  - `deactivatedFailureIds` string[]: IDs of failure injections that were deactivated as part of the recovery

## Other responses

- `400`: Resource not found, already healthy, or neither resourceName nor resourceId supplied
- `402`: Payment required (x402)
- `403`: Access denied (not the owner)
- `404`: Simulation not found

---

[API](https://skmtc.dev/cloudworldmodel/apis/cloud-world-model-agent-api.md) · [All operations](https://skmtc.dev/cloudworldmodel/apis/cloud-world-model-agent-api/llms.txt) · [OpenAPI document](https://skmtc.dev/cloudworldmodel/apis/cloud-world-model-agent-api/revisions/ecc3ad1ff07f?raw)