Sign in
Internal

Set the deployment pool's load-shedding override (platform admin)

Replaces the endpoint pool's whole load-shedding override (PoolShedding). Every field zero-value = inherit the global/env behavior, so an all-empty body clears the override. This edits the POOL (deployments sharing a pool share it) and reaches every gateway task in ~2s via the config generation stamp; on a subsequent revise the new generation's pool inherits it, so set it on the CURRENT generation to have the next one carry it forward. Returns the refreshed replica view.

post/v1/admin/inference/deployments/{id}/pool-shedding

Path parameters

idstring required

Request body

brownoutTtftP95Secinteger

Arms the pool brownout breaker at this rolling-TTFT-p95 threshold; 0 = disabled (the default).

concurrencyMode'' | 'enforce' | 'shadow'

Overrides global concurrency enforcement for this pool's member caps. "enforce" makes the fleet counter decide, "shadow" keeps the per-task ceiling deciding, "" (or absent) inherits the global decision.

exclusionMode'' | 'enforce' | 'shadow'

Overrides global member-health exclusion for this pool. "enforce" removes failing members whatever the global env says, "shadow" only records, "" (or absent) inherits the global decision.

firstAttemptTtftSecinteger

Caps attempt 1 of a streaming request when the plan has >= 2 dial targets, so one failover stays affordable. 0 = inherit the global default; capped at half the effective TTFT budget by the gateway.

maxExcludedFractionnumber double

Panic threshold: health exclusion may never remove more than this share of active members at once. Must stay in [0,1) (the exclusive upper bound is enforced by the server); 0 = the gateway default (0.5).

ttftBudgetSecinteger

Per-request first-token budget for this pool's streaming attempts; 0 = the global default.

Response

Updated

affinityGuardUtilnumber double

prefix_affinity overflow bound; absent = the gateway default

policy'p2c_util' | 'prefix_affinity' required

The pool's member-selection policy (p2c_util when the pool row does not exist yet)

poolIdstring required

The deployment's endpoint pool id

strayMembersstring[]

Pool member hostnames no current replica owns (pending the reconcile loop's stray sweep). Usually empty.

Changes

Changed in 1 of the 19 revisions of this API.1