Sign in
sandboxes

Read Runaway

Running boxes past their class threshold, worst overage first.

Thresholds are keyed by (provider, gpu_type) in deployment config, not by a constant here: an eight-way H100 host running four hours is a normal job, and a CPU box running four days is a leak.

Ranked on each box's current uninterrupted run rather than its lifetime running total, and restricted to boxes that are running. See :func:load_runaway_boxes for why either alone makes the view useless.

Ordering carries the ranking, and the key is the overage -- the run length minus that box's own budget -- not the run length. Because budgets differ per class, the two orders differ: a CPU box nine hours into a one-hour budget outranks a GPU box twelve hours into a ten-hour one, and so appears above a row with a longer currentRunSeconds.

Neither the overage nor the threshold is on the wire, so a reader cannot reconstruct the ordering from the rows; it has to be taken on trust, and currentRunSeconds is context for a row rather than proof of its position. Putting the overage on the response would make it checkable, at the cost of a field the shared contract does not have.

Truncation is reported in X-Runaway-Truncated rather than swallowed: a runaway view that silently drops rows reads as "nothing else is wrong" when the opposite may be true. The header covers both places rows are dropped -- the candidate cap and this route's limit -- because a fleet is far likelier to have more than limit boxes over threshold than more than MAX_RUNAWAY_CANDIDATES boxes running.

get/api/sandboxes/admin/runaway

Query parameters

providerSandboxProvider[]
limitinteger

Response

Successful Response

provider'modal' | 'aws' | 'gcp' | 'sail' required

The compute backend that owns a sandbox.

sandboxIdstring required
sandboxAliasstring required
lifecyclePhase'pending' | 'running' | 'stopped' | 'deleted' required

Current phase of a sandbox in the ledger's state table.

STOPPED is not DELETED. checkpoint_and_stop retains the disk and is the normal cleanup path on GCP and Sail, so a stopped sandbox is still billing for storage.

RUNNING and STOPPED are therefore the two "open" phases: a sandbox in either one still exists and still costs money. The state table's partial unique index on sandbox_alias is scoped to exactly this pair, so that an alias resolves to at most one open sandbox. Adding or removing an open phase means changing that index in the same migration.

warmth'cold' | 'warm_provider' | 'warm_gateway' required

How much of a sandbox acquisition was served from something existing.

Three values, not two. Provider-level reuse and gateway session reuse are separate events raised from separate call sites, and they have different latency profiles; collapsing them into one "warm" value loses the distinction the cold-start work depends on.

resourceClassstring required
gpuTypestring nullable required
gpuCountinteger nullable required
isPackedboolean required
isIsolatedboolean required
commitmentBasis'on_demand' | 'reservation' | 'capacity_block' required

Which capacity pool a sandbox's launch actually consumed.

A property of the individual launch, not of the resource class. The same shape lands differently from one start to the next: GCP GPU hosts launch into GCP_GPU_RESERVATION "only while the reservation has an idle slot" (gcp_client.py:32-35) and take the on-demand path when it is unset (:166), and an AWS capacity-block launch "falls back to the on-demand path instead of failing the kernel start" when the slot is raced away (ec2_client.py:1268).

So two identical p5.48xlarge rows can carry completely different real cost, and nothing at read time can tell them apart — the pool a launch consumed is unrecoverable once the box is gone. That is what makes this a stored column rather than something the rate card derives.

RESERVATION and CAPACITY_BLOCK are separate because the mechanisms are: a GCE specific reservation holds RESERVATION_BOUND VMs, while AWS capacity blocks are bought on a rolling cadence and reclaimed at expiry (QUA-2813 and QUA-2770 respectively). Neither is marginal spend.

deploymentEnvironmentstring required
notebookIdstring nullable required
kernelIdstring nullable required
orgIdstring nullable required
orgNamestring nullable
workspaceIdstring nullable required
userIdstring nullable required
userEmailstring nullable
runScopestring nullable required
createdAtstring required
lastTransitionAtstring required
causeOfDeath'idle_reap' | 'preemption' | 'health_check_kill' | 'readiness_failed' | 'lease_released' | 'lease_expired' | 'age_limit' | 'explicit_teardown' | 'superseded' | 'unknown_reconciled' required

Why a sandbox left its last open phase.

UNKNOWN_RECONCILED is first-class and load-bearing. A hard preemption leaves nobody running to write a truthful cause, so the reconciler has to be able to say it does not know. Without this value the reconciler is forced to pick a plausible cause instead, and every other cause in the ledger stops being trustworthy.

closedAtstring nullable required
runningSecondsinteger nullable required
stoppedSecondsinteger nullable required
currentRunSecondsinteger nullable required

Changes

Changed in 3 of the 30 revisions of this API.45

  • abf76937a75142See the full diff
    • ●

      added the new age_limit enum value to the // response property for the response status

      response-property-enum-value-added

    • ●

      added the new lease_expired enum value to the // response property for the response status

      response-property-enum-value-added

    • ●

      added the new lease_released enum value to the // response property for the response status

      response-property-enum-value-added

    • ●

      added the new readiness_failed enum value to the // response property for the response status

      response-property-enum-value-added

    • ○

      added the optional property // to the response with the status

      response-optional-property-added

    • ○

      added the optional property // to the response with the status

      response-optional-property-added

    This revision also has 3 changes that name no endpoint, such as unreferenced schemas being removed. See the revision's changelog

  • 21910d3ae5d82See the full diff
    • ○

      added the optional property / to the response with the status

      response-optional-property-added

    • ○

      added the optional property / to the response with the status

      response-optional-property-added

  • 3a8d4a28ae931See the full diff
    • ○

      endpoint added

      endpoint-added