---
title: "Create a self-hosted serving deployment (platform admin)"
method: POST
path: "/v1/admin/inference/deployments"
tags: ["Internal"]
---

# Create a self-hosted serving deployment (platform admin)

`POST /v1/admin/inference/deployments`

Records the operator's intent to run N replicas of an inference engine for one public catalog model. The spec is validated hard at admission (the engine must exist and the replica must plan) so a row that could never dispatch is rejected here instead of failing later. The new deployment starts in status pending; the serving reconcile loop admits replicas, stages weights, probes readiness, and registers the pool backend from there.

## Request body

- CreateServingDeploymentRequest
  - `backendWeight` integer — Initial routing weight; defaults to 0 (standby)
  - `dtype` string
  - `engine` string, required — Must resolve to a known engine (e.g. vllm, sglang)
  - `engineImage` string — Engine container image override
  - `env` object
  - `extraArgs` string[]
  - `gpuCountPerReplica` integer, required
  - `gpuModelId` string, required
  - `isolated` boolean — Launch and health-check replicas without ever publishing their pool as a catalog backend. Requires backendWeight 0.
  - `maxConcurrencyPerReplica` integer
  - `maxModelLen` integer
  - `maxPromptBytes` integer — Exclusive text-request byte ceiling for this deployment's routing lane; 0 or absent = unbounded
  - `minPromptBytes` integer — Inclusive text-request byte floor for this deployment's routing lane; 0 or absent = unbounded
  - `nativeFamilies` string[] — Defaults to chat_completions at registration
  - `nodePool` string — Stable logical capacity lane, e.g. rtx5090 or h100. Deployments in different node pools keep independent routing weights.
  - `organizationId` string, required — Org the managed replicas run (and meter) under
  - `publicModelId` string, required — Catalog model the pool backend registers on. Must already be seeded, with pricing
  - `quantization` string
  - `replicasDesired` integer, required — Must lie within replicasMin..replicasMax
  - `replicasMax` integer, required
  - `replicasMin` integer, required
  - `servedModelId` string, required — Must equal the catalog backend ref the gateway rewrites to
  - `version` string
  - `weightsRef` string — Staged-checkpoint reference; omit to let the engine load from its own source

## Response `200`

Created

- ServingDeployment — A self-hosted serving deployment: the durable intent to run N replicas of an inference engine for one public catalog model, plus the runtime state the reconcile loop maintains while converging on it.
  - `backendCatalogWeight` integer — Live catalog routing weight of this deployment's backend. Only present when backendRegistered is true. May differ from backendWeight (the row mirror) while a rollout ramp is in flight.
  - `backendHealthy` boolean — Live catalog health of this deployment's backend (the kill switch). Only present when backendRegistered is true.
  - `backendRegistered` boolean — Whether this deployment's pool:// backend is currently present on the catalog model row. Absent when the live catalog state could not be read; false until the first replica passes readiness.
  - `backendWeight` integer, required — Routing weight of the catalog backend. 0 = standby (blue/green starts here)
  - `createdAt` string, required
  - `dtype` string
  - `engine` string, required — Inference engine, e.g. vllm or sglang
  - `engineImage` string — Engine container image override; absent = the engine's default
  - `env` object
  - `extraArgs` string[]
  - `gpuCountPerReplica` integer, required
  - `gpuModelId` string, required
  - `id` string, required
  - `isolated` boolean, required — True when replicas and their private endpoint pool are for direct operator testing only. Isolated deployments never register a catalog backend, never participate in blue/green rollout, and cannot receive gateway traffic.
  - `maxConcurrencyPerReplica` integer
  - `maxModelLen` integer
  - `maxPromptBytes` integer — Exclusive text-request byte ceiling for routing eligibility; 0 or absent = unbounded. Routing hint only, not a context limit.
  - `minPromptBytes` integer — Inclusive text-request byte floor for routing eligibility; 0 or absent = unbounded. Routing hint only, not a context limit.
  - `nativeFamilies` string[]
  - `nodePool` string — Stable logical capacity lane within the public model. Blue/green only replaces versions in the same node pool; empty preserves legacy model-wide grouping.
  - `organizationId` string, required — Org that owns the managed replicas (and their metering)
  - `poolId` string — Endpoint pool the deployment's replicas register in
  - `publicModelId` string, required — Catalog model this deployment serves capacity for
  - `quantization` string
  - `readyReplicas` integer, required
  - `replicaVmIds` string[]
  - `replicasDesired` integer, required
  - `replicasMax` integer, required
  - `replicasMin` integer, required
  - `servedModelId` string, required — Model id the engine serves; must equal the catalog backend ref
  - `status` 'pending' | 'deploying' | 'ready' | 'degraded' | 'scaling' | 'draining' | 'stopped' | 'failed', required
  - `statusReason` string — Human-readable cause for degraded/failed states
  - `updatedAt` string, required
  - `version` string — Generation marker for blue/green within one public model and node pool
  - `weightsRef` string — Staged-checkpoint reference; absent = the engine loads from its own source

## Other responses

- `400` — The request is invalid
- `401` — Missing or invalid API key
- `403` — API key lacks the required scope
- `503` — A required integration (e.g. payments) is not configured

---

[API](https://skmtc.dev/openrelay/apis/openrelay-api.md) · [All operations](https://skmtc.dev/openrelay/apis/openrelay-api/llms.txt) · [OpenAPI document](https://skmtc-service-production.skmtc.workers.dev/v1/apis/openrelay/openrelay-api/revisions/4c42fd61ed41/schema)