Sign in
Endpoints

Create a dedicated endpoint

Creates a new dedicated endpoint for serving models. The endpoint starts automatically after creation. You can deploy any supported model on hardware configurations that meet the model's requirements.

post/endpoints

Request body

display_namestring

A human-readable name for the endpoint

modelstring required

The model to deploy on this endpoint

hardwarestring required

The hardware configuration to use for this endpoint

disable_prompt_cacheboolean

This parameter is deprecated and no longer has any effect.

disable_speculative_decodingboolean

Whether to disable speculative decoding for this endpoint

state'STARTED' | 'STOPPED'

The desired state of the endpoint

inactive_timeoutinteger nullable

The number of minutes of inactivity after which the endpoint stops automatically. Set to null, omit, or set to 0 to disable automatic timeout.

availability_zonestring

Create the endpoint in a specified availability zone (e.g., us-central-4b)

Example request

{
  "display_name": "My Llama3 70b endpoint",
  "model": "deepseek-ai/DeepSeek-R1",
  "hardware": "1x_nvidia_a100_80gb_sxm",
  "autoscaling": {
    "min_replicas": 2,
    "max_replicas": 5
  },
  "state": "STARTED",
  "inactive_timeout": 60
}

Response

200

object'endpoint' required

The object type, which is always endpoint.

idstring required

Unique identifier for the endpoint

namestring required

System name for the endpoint

display_namestring required

Human-readable name for the endpoint

modelstring required

The model deployed on this endpoint

hardwarestring required

The hardware configuration used for this endpoint

type'dedicated' required

The type of endpoint

ownerstring required

The owner of this endpoint

state'PENDING' | 'STARTING' | 'STARTED' | 'STOPPING' | 'STOPPED' | 'ERROR' required

Current state of the endpoint

created_atstring date-time required

Timestamp when the endpoint was created

Example response

{
  "id": "endpoint-d23901de-ef8f-44bf-b3e7-de9c1ca8f2d7",
  "name": "devuser/deepseek-ai/DeepSeek-R1-a32b82a1",
  "display_name": "My DeepSeek R1 endpoint",
  "model": "deepseek-ai/DeepSeek-R1",
  "hardware": "8x_nvidia_h200_140gb_sxm",
  "type": "dedicated",
  "owner": "devuser",
  "state": "STARTED",
  "autoscaling": {
    "min_replicas": 2,
    "max_replicas": 5
  },
  "created_at": "2025-02-04T10:43:55.405Z"
}

Changes

Changed in 1 of the 14 revisions of this API.22

  • 0c4fdad7743522See the full diff
    • ▲

      the autoscaling/max_replicas response's property type/format changed from integer/int32 to integer/ for status 200

      response-property-type-changed

    • ▲

      the autoscaling/min_replicas response's property type/format changed from integer/int32 to integer/ for status 200

      response-property-type-changed

    • ○

      the autoscaling/max_replicas request property type/format was generalized from integer/int32 to integer/

      request-property-type-generalized

    • ○

      the autoscaling/min_replicas request property type/format was generalized from integer/int32 to integer/

      request-property-type-generalized

    This revision also has 2 changes that name no endpoint, such as unreferenced schemas being removed. See the revision's changelog