---
title: "Set the deployment pool's member-selection routing (platform admin)"
method: POST
path: "/v1/admin/inference/deployments/{id}/pool-routing"
tags: ["Internal"]
---

# Set the deployment pool's member-selection routing (platform admin)

`POST /v1/admin/inference/deployments/{id}/pool-routing`

Sets the endpoint pool's member-selection policy and/or the prefix_affinity load guard. This edits the POOL (deployments sharing a pool share the setting) and reaches every gateway task in ~2s via the config generation stamp; setting policy back to p2c_util is the instant rollback. Returns the refreshed replica view.

## Path parameters

- `id` string, required

## Request body

- ServingPoolRoutingRequest — Pool routing knobs; only the provided fields change. policy switches the member-selection algorithm; affinityGuardUtil bounds how full a conversation's home member may run before prefix_affinity overflows to load-only P2C (0 clears it back to the gateway default).
  - `affinityGuardUtil` number, double
  - `policy` 'p2c_util' | 'prefix_affinity'

## Response `200`

Updated

- ServingReplicaList
  - `affinityGuardUtil` number, double — prefix_affinity overflow bound; absent = the gateway default
  - `items` ServingReplica[], required
    - `capacity` integer — In-flight bound the engine was launched with (= pool member capacity)
    - `createdAt` string
    - `drainHold` boolean — An operator has taken this replica out of rotation. The reconcile loop keeps its pool member draining while set, instead of re-activating it once it is ready again.
    - `drainHoldReason` string — Why the replica was held out of rotation
    - `gpuCount` integer
    - `hostname` string — run.* hostname the gateway dials; the pool-member key
    - `load` ServingReplicaLoad — The replica's last scraped engine load, as published to the gateways. Absent fields mean the engine did not expose the metric; a fabricated zero is never reported.
      - `atMs` integer, required — Publish time, unix milliseconds; readings expire after ~45s
      - `hasKvUtil` boolean
      - `hasWaiting` boolean
      - `kvUtil` number — KV-cache utilization 0..1; present only when hasKvUtil
      - `running` integer, required — Requests currently executing on the engine
      - `waiting` integer — Requests queued on the engine; present only when hasWaiting
    - `memberCapacity` integer — Capacity currently advertised on the pool member; present only when a member exists
    - `memberState` 'active' | 'draining' | 'out' | 'none', required — Pool membership: none = not (yet) a member of the deployment's pool
    - `memberWeight` number, double — Ranking weight on the pool member (the warmup/canary dial); absent = full share (1.0)
    - `name` string
    - `nodeId` string — Node the replica placed on; absent before placement
    - `nodeName` string — Memorable display label for nodeId (adjective-noun-hex); absent before placement
    - `nomadAllocId` string
    - `provisioningStage` string
    - `runtime` 'vm' | 'pod' — What backs this replica. A LABEL only, derived from the replica row's capability tier: every operator action here is runtime-neutral, because a pool member is a hostname and the pool does not know what serves it.
    - `status` string, required — VM lifecycle status (deploying, running, stopped, terminated, failed); "unreadable" when the row could not be read this request
    - `statusReason` string — Defer/failure reason from dispatch (weights staging, image staging, placement, ...)
    - `vmId` string, required
  - `policy` 'p2c_util' | 'prefix_affinity', required — The pool's member-selection policy (p2c_util when the pool row does not exist yet)
  - `poolId` string, required — The deployment's endpoint pool id
  - `shedding` ServingPoolShedding — The pool's OPTIONAL load-shedding override, set and read as one whole block. Every field zero-value (or absent) = inherit the gateway's global/env behavior, so an all-empty block is "no override": setting one clears the override back to inherit. As a request body this REPLACES the pool's whole block, so send every field the pool should keep, not just the one changed. Edits the POOL (deployments sharing a pool share it) and reach every gateway task in ~2s via the config generation stamp. Default-off is unchanged: a deployment that never sets this keeps today's exact behavior.
    - `brownoutTtftP95Sec` integer — Arms the pool brownout breaker at this rolling-TTFT-p95 threshold; 0 = disabled (the default).
    - `concurrencyMode` '' | 'enforce' | 'shadow' — Overrides global concurrency enforcement for this pool's member caps. "enforce" makes the fleet counter decide, "shadow" keeps the per-task ceiling deciding, "" (or absent) inherits the global decision.
    - `exclusionMode` '' | 'enforce' | 'shadow' — Overrides global member-health exclusion for this pool. "enforce" removes failing members whatever the global env says, "shadow" only records, "" (or absent) inherits the global decision.
    - `firstAttemptTtftSec` integer — Caps attempt 1 of a streaming request when the plan has >= 2 dial targets, so one failover stays affordable. 0 = inherit the global default; capped at half the effective TTFT budget by the gateway.
    - `maxExcludedFraction` number, double — Panic threshold: health exclusion may never remove more than this share of active members at once. Must stay in [0,1) (the exclusive upper bound is enforced by the server); 0 = the gateway default (0.5).
    - `ttftBudgetSec` integer — Per-request first-token budget for this pool's streaming attempts; 0 = the global default.
  - `strayMembers` string[] — Pool member hostnames no current replica owns (pending the reconcile loop's stray sweep). Usually empty.

## Other responses

- `400` — The request is invalid
- `401` — Missing or invalid API key
- `403` — API key lacks the required scope
- `404` — Resource not found
- `409` — The request conflicts with existing state
- `503` — A required integration (e.g. payments) is not configured

## Changes

- **2026-08-29** `3dc47c9667e8` — 1 info
  - added the optional property `shedding` to the response with the `200` status
- **2026-08-26** `9dad68d3afa1` — 1 info
  - added the optional property `items/items/nodeName` to the response with the `200` status
- **2026-08-16** `ed76b41e8006` — 3 info
  - added the optional property `items/items/drainHold` to the response with the `200` status
  - added the optional property `items/items/drainHoldReason` to the response with the `200` status
  - added the optional property `items/items/runtime` to the response with the `200` status

[Change history](https://skmtc.dev/openrelay/apis/openrelay-api/changes/v1/admin/inference/deployments/:id/pool-routing/post.md)

---

[API](https://skmtc.dev/openrelay/apis/openrelay-api.md) · [All operations](https://skmtc.dev/openrelay/apis/openrelay-api/llms.txt) · [OpenAPI document](https://skmtc-service-production.skmtc.workers.dev/v1/apis/openrelay/openrelay-api/revisions/3dc47c9667e8/schema)