--- title: "Read Runaway" method: GET path: "/api/sandboxes/admin/runaway" tags: ["sandboxes"] --- # Read Runaway `GET /api/sandboxes/admin/runaway` Running boxes past their class threshold, worst overage first. Thresholds are keyed by ``(provider, gpu_type)`` in deployment config, not by a constant here: an eight-way H100 host running four hours is a normal job, and a CPU box running four days is a leak. Ranked on each box's current uninterrupted run rather than its lifetime running total, and restricted to boxes that are running. See :func:`load_runaway_boxes` for why either alone makes the view useless. Ordering carries the ranking, and the key is the **overage** -- the run length minus that box's own budget -- not the run length. Because budgets differ per class, the two orders differ: a CPU box nine hours into a one-hour budget outranks a GPU box twelve hours into a ten-hour one, and so appears above a row with a longer ``currentRunSeconds``. Neither the overage nor the threshold is on the wire, so a reader cannot reconstruct the ordering from the rows; it has to be taken on trust, and ``currentRunSeconds`` is context for a row rather than proof of its position. Putting the overage on the response would make it checkable, at the cost of a field the shared contract does not have. Truncation is reported in ``X-Runaway-Truncated`` rather than swallowed: a runaway view that silently drops rows reads as "nothing else is wrong" when the opposite may be true. The header covers both places rows are dropped -- the candidate cap and this route's ``limit`` -- because a fleet is far likelier to have more than ``limit`` boxes over threshold than more than ``MAX_RUNAWAY_CANDIDATES`` boxes running. ## Query parameters - `provider` SandboxProvider[] - `limit` integer ## Response `200` Successful Response - SandboxRow[] - `provider` 'modal' | 'aws' | 'gcp' | 'sail', required — The compute backend that owns a sandbox. - `sandboxId` string, required - `sandboxAlias` string, required - `lifecyclePhase` 'pending' | 'running' | 'stopped' | 'deleted', required — Current phase of a sandbox in the ledger's state table. ``STOPPED`` is not ``DELETED``. ``checkpoint_and_stop`` retains the disk and is the normal cleanup path on GCP and Sail, so a stopped sandbox is still billing for storage. ``RUNNING`` and ``STOPPED`` are therefore the two "open" phases: a sandbox in either one still exists and still costs money. The state table's partial unique index on ``sandbox_alias`` is scoped to exactly this pair, so that an alias resolves to at most one open sandbox. Adding or removing an open phase means changing that index in the same migration. - `warmth` 'cold' | 'warm_provider' | 'warm_gateway', required — How much of a sandbox acquisition was served from something existing. Three values, not two. Provider-level reuse and gateway session reuse are separate events raised from separate call sites, and they have different latency profiles; collapsing them into one "warm" value loses the distinction the cold-start work depends on. - `resourceClass` string, required - `gpuType` string, nullable, required - `gpuCount` integer, nullable, required - `isPacked` boolean, required - `isIsolated` boolean, required - `commitmentBasis` 'on_demand' | 'reservation' | 'capacity_block', required — Which capacity pool a sandbox's launch actually consumed. A property of the individual launch, not of the resource class. The same shape lands differently from one start to the next: GCP GPU hosts launch into ``GCP_GPU_RESERVATION`` "only while the reservation has an idle slot" (``gcp_client.py:32-35``) and take the on-demand path when it is unset (``:166``), and an AWS capacity-block launch "falls back to the on-demand path instead of failing the kernel start" when the slot is raced away (``ec2_client.py:1268``). So two identical ``p5.48xlarge`` rows can carry completely different real cost, and nothing at read time can tell them apart — the pool a launch consumed is unrecoverable once the box is gone. That is what makes this a stored column rather than something the rate card derives. ``RESERVATION`` and ``CAPACITY_BLOCK`` are separate because the mechanisms are: a GCE specific reservation holds ``RESERVATION_BOUND`` VMs, while AWS capacity blocks are bought on a rolling cadence and reclaimed at expiry (QUA-2813 and QUA-2770 respectively). Neither is marginal spend. - `deploymentEnvironment` string, required - `notebookId` string, nullable, required - `kernelId` string, nullable, required - `orgId` string, nullable, required - `orgName` string, nullable - `workspaceId` string, nullable, required - `userId` string, nullable, required - `userEmail` string, nullable - `runScope` string, nullable, required - `createdAt` string, required - `lastTransitionAt` string, required - `causeOfDeath` 'idle_reap' | 'preemption' | 'health_check_kill' | 'readiness_failed' | 'lease_released' | 'lease_expired' | 'age_limit' | 'explicit_teardown' | 'superseded' | 'unknown_reconciled', required — Why a sandbox left its last open phase. ``UNKNOWN_RECONCILED`` is first-class and load-bearing. A hard preemption leaves nobody running to write a truthful cause, so the reconciler has to be able to say it does not know. Without this value the reconciler is forced to pick a plausible cause instead, and every other cause in the ledger stops being trustworthy. - `closedAt` string, nullable, required - `runningSeconds` integer, nullable, required - `stoppedSeconds` integer, nullable, required - `currentRunSeconds` integer, nullable, required - `cost` union, required - CostAvailable — A derived cost floor for one box, split by the kind of money it is. Three figures in three different units, and **none may be added to another**. The rate module refuses to produce a total and this shape keeps that refusal, because summing prepaid capacity with marginal spend produces a figure no invoice will ever show. - ``on_demand_usd_floor`` is money that left the account, underestimated. - ``committed_capacity_list_value_usd`` is what the capacity this box drew would have cost at list price had it not been prepaid. Not spend: the slot was paid for whether or not anything ran in it. - ``consumed_capacity_hours`` is slot-hours drawn from a commitment. It needs no rate card, so it is the one figure here that cannot be wrong. ``basis`` names the pool the box's *compute* drew on, which is what a dashboard groups by. It does not promise every dollar below it is that kind; the fields do that. - `available` true - `basis` 'on_demand' | 'committed', required - `onDemandUsdFloor` string, nullable, required - `committedCapacityListValueUsd` string, nullable, required - `consumedCapacityHours` string, nullable, required - `billingShape` string, required - `rateProvenance` 'vendor_list' | 'vendor_reserved', required - `rateRegion` string, nullable, required - `rateRevisedOn` string, required - `overheadMultiplier` string, required - CostUnavailable — No derivable figure, and the reason there is none. Every reason is a property of one box rather than of the deployment, and every one of them yields no dollars at all. None is a zero: a zero renders as free, and nothing in the ledger is free. Three of the four originate in ``compute.sandbox_cost`` and are passed through rather than collapsed. A Sail box (``usage_billed_provider``) and a box whose machine type the card has no entry for (``unpriced_shape``) are different problems with different fixes, and a reader that has to act on one should not have to guess which it is looking at. The fourth, ``no_duration_evidence``, is this module's own: it is about the event log rather than the rate card. ``unknown_resource_class`` arrives by both routes. The rate module returns it for a shape it cannot resolve; the read layer returns it as well, for a dedicated box whose ``gpuCount`` is null -- unresolvable for the different reason that the hardware was never learned. Those have different fixes, a rate-card entry against a reconciler backfill, but they do not need separate reasons, because the row already carries the discriminator: this reason beside a null ``gpuCount`` is the second case, and every other occurrence is the first. - `available` false - `reason` 'usage_billed_provider' | 'unpriced_shape' | 'unknown_resource_class' | 'no_duration_evidence', required ## Other responses - `422` — Validation Error ## Changes - **2026-09-18** `abf76937a751` — 4 warning, 2 info - added the new `age_limit` enum value to the `items/causeOfDeath/anyOf[subschema #1: CauseOfDeath]/` response property for the response status `200` - added the new `lease_expired` enum value to the `items/causeOfDeath/anyOf[subschema #1: CauseOfDeath]/` response property for the response status `200` - added the new `lease_released` enum value to the `items/causeOfDeath/anyOf[subschema #1: CauseOfDeath]/` response property for the response status `200` - added the new `readiness_failed` enum value to the `items/causeOfDeath/anyOf[subschema #1: CauseOfDeath]/` response property for the response status `200` - …2 more - **2026-09-15** `21910d3ae5d8` — 2 info - added the optional property `items/orgName` to the response with the `200` status - added the optional property `items/userEmail` to the response with the `200` status - **2026-09-10** `3a8d4a28ae93` — 1 info - endpoint added [Change history](https://skmtc.dev/quadrillion/apis/quadrillion-cloud-api/changes/api/sandboxes/admin/runaway/get.md) --- [API](https://skmtc.dev/quadrillion/apis/quadrillion-cloud-api.md) · [All operations](https://skmtc.dev/quadrillion/apis/quadrillion-cloud-api/llms.txt) · [OpenAPI document](https://skmtc.dev/quadrillion/apis/quadrillion-cloud-api/revisions/397ad3a1d858?raw)