---
title: "Per-replica runtime view of a serving deployment (platform admin)"
method: GET
path: "/v1/admin/inference/deployments/{id}/replicas"
tags: ["Internal"]
---

# Per-replica runtime view of a serving deployment (platform admin)

`GET /v1/admin/inference/deployments/{id}/replicas`

Joins each replica VM row with its pool-member state and the last scraped engine load, so an operator can see where every replica is stuck (placement, weights staging, image staging, readiness) and what traffic it is taking, without walking Nomad and Redis by hand.

## Path parameters

- `id` string, required

## Response `200`

The deployment's replicas

- ServingReplicaList
  - `affinityGuardUtil` number, double — prefix_affinity overflow bound; absent = the gateway default
  - `items` ServingReplica[], required
    - `capacity` integer — In-flight bound the engine was launched with (= pool member capacity)
    - `createdAt` string
    - `drainHold` boolean — An operator has taken this replica out of rotation. The reconcile loop keeps its pool member draining while set, instead of re-activating it once it is ready again.
    - `drainHoldReason` string — Why the replica was held out of rotation
    - `gpuCount` integer
    - `hostname` string — run.* hostname the gateway dials; the pool-member key
    - `load` ServingReplicaLoad — The replica's last scraped engine load, as published to the gateways. Absent fields mean the engine did not expose the metric; a fabricated zero is never reported.
      - `atMs` integer, required — Publish time, unix milliseconds; readings expire after ~45s
      - `hasKvUtil` boolean
      - `hasWaiting` boolean
      - `kvUtil` number — KV-cache utilization 0..1; present only when hasKvUtil
      - `running` integer, required — Requests currently executing on the engine
      - `waiting` integer — Requests queued on the engine; present only when hasWaiting
    - `memberCapacity` integer — Capacity currently advertised on the pool member; present only when a member exists
    - `memberState` 'active' | 'draining' | 'out' | 'none', required — Pool membership: none = not (yet) a member of the deployment's pool
    - `memberWeight` number, double — Ranking weight on the pool member (the warmup/canary dial); absent = full share (1.0)
    - `name` string
    - `nodeId` string — Node the replica placed on; absent before placement
    - `nodeName` string — Memorable display label for nodeId (adjective-noun-hex); absent before placement
    - `nomadAllocId` string
    - `provisioningStage` string
    - `runtime` 'vm' | 'pod' — What backs this replica. A LABEL only, derived from the replica row's capability tier: every operator action here is runtime-neutral, because a pool member is a hostname and the pool does not know what serves it.
    - `status` string, required — VM lifecycle status (deploying, running, stopped, terminated, failed); "unreadable" when the row could not be read this request
    - `statusReason` string — Defer/failure reason from dispatch (weights staging, image staging, placement, ...)
    - `vmId` string, required
  - `policy` 'p2c_util' | 'prefix_affinity', required — The pool's member-selection policy (p2c_util when the pool row does not exist yet)
  - `poolId` string, required — The deployment's endpoint pool id
  - `shedding` ServingPoolShedding — The pool's OPTIONAL load-shedding override, set and read as one whole block. Every field zero-value (or absent) = inherit the gateway's global/env behavior, so an all-empty block is "no override": setting one clears the override back to inherit. As a request body this REPLACES the pool's whole block, so send every field the pool should keep, not just the one changed. Edits the POOL (deployments sharing a pool share it) and reach every gateway task in ~2s via the config generation stamp. Default-off is unchanged: a deployment that never sets this keeps today's exact behavior.
    - `brownoutTtftP95Sec` integer — Arms the pool brownout breaker at this rolling-TTFT-p95 threshold; 0 = disabled (the default).
    - `concurrencyMode` '' | 'enforce' | 'shadow' — Overrides global concurrency enforcement for this pool's member caps. "enforce" makes the fleet counter decide, "shadow" keeps the per-task ceiling deciding, "" (or absent) inherits the global decision.
    - `exclusionMode` '' | 'enforce' | 'shadow' — Overrides global member-health exclusion for this pool. "enforce" removes failing members whatever the global env says, "shadow" only records, "" (or absent) inherits the global decision.
    - `firstAttemptTtftSec` integer — Caps attempt 1 of a streaming request when the plan has >= 2 dial targets, so one failover stays affordable. 0 = inherit the global default; capped at half the effective TTFT budget by the gateway.
    - `maxExcludedFraction` number, double — Panic threshold: health exclusion may never remove more than this share of active members at once. Must stay in [0,1) (the exclusive upper bound is enforced by the server); 0 = the gateway default (0.5).
    - `ttftBudgetSec` integer — Per-request first-token budget for this pool's streaming attempts; 0 = the global default.
  - `strayMembers` string[] — Pool member hostnames no current replica owns (pending the reconcile loop's stray sweep). Usually empty.

## Other responses

- `401` — Missing or invalid API key
- `403` — API key lacks the required scope
- `404` — Resource not found
- `503` — A required integration (e.g. payments) is not configured

---

[API](https://skmtc.dev/openrelay/apis/openrelay-api.md) · [All operations](https://skmtc.dev/openrelay/apis/openrelay-api/llms.txt) · [OpenAPI document](https://skmtc-service-production.skmtc.workers.dev/v1/apis/openrelay/openrelay-api/revisions/3dc47c9667e8/schema)