Chat Completions

Openai Chat Completions

Changed on
post/v1/chat/completions

Request

  • Base URL: https://api.deepinfra.com
  • URL: https://api.deepinfra.com/v1/chat/completions
  • Auth: HTTP bearer

Headers

x-deepinfra-sourcestring nullable
x-deepinfra-service-tierstring nullable

Per-request service tier (priority or flex) for clients that cannot set the service_tier body field. The body field wins when both are present; unrecognized values ride the default tier.

xi-api-keystring nullable
x-api-keystring nullable

Request body

service_tier'default' | 'priority' | 'flex'
fail_fastboolean

If true, the request is rejected immediately with HTTP 429 when the model has no spare capacity, instead of waiting in the queue. Opt-in; the default (false) keeps standard queueing behavior.

modelsstring[] nullable

Ordered list of up to 4 fallback models. The request is attempted on each model in order: when a model rejects it for lack of capacity (HTTP 429 model-busy / flex no-capacity), the next model is tried server-side. The first model that accepts serves the request; the response's model field and billing reflect that model, at that model's pricing. Models before the last are attempted without queueing (as if fail_fast were set); the last model honors the request's own fail_fast value. When models is set, the model field is ignored. Entries must be plain model names (no deploy_id:, custom_hostport, or :revision specifiers); duplicate entries are ignored, keeping the first occurrence.

modelstring required

model name

streamboolean

whether to stream the output via SSE or return the full response

temperaturenumber

What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic

top_pnumber

An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens comprising the top 10% probability mass are considered.

min_pnumber

Float that represents the minimum probability for a token to be considered, relative to the probability of the most likely token. Must be in [0, 1]. Set to 0 to disable this.

top_kinteger

Sample from the best k (number of) tokens. 0 means off

max_tokensinteger nullable

The maximum number of tokens to generate in the chat completion.

The total length of input tokens and generated tokens is limited by the model's context length. If explicitly set to None it will be the model's max context length minus input length or 65536, whichever is smaller.

stop_token_idsinteger[] nullable

Up to 16 token IDs where the API will stop generating further tokens. Merged with the model's built-in stop tokens. Intended for private deployments.

ninteger

number of sequences to return

presence_penaltynumber

Positive values penalize new tokens based on whether they appear in the text so far, increasing the model's likelihood to talk about new topics.

frequency_penaltynumber

Positive values penalize new tokens based on how many times they appear in the text so far, increasing the model's likelihood to talk about new topics.

repetition_penaltynumber

Alternative penalty for repetition, but multiplicative instead of additive (> 1 penalize, < 1 encourage)

userstring nullable

A unique identifier representing your end-user, which can help monitor and detect abuse. Avoid sending us any identifying information. We recommend hashing user identifiers.

seedinteger nullable

Seed for random number generator. If not provided, a random seed is used. Determinism is not guaranteed.

logprobsboolean nullable

Whether to return log probabilities of the output tokens or not.If true, returns the log probabilities of each output token returned in the content of message.

reasoning_effort'none' | 'minimal' | 'low' | 'medium' | 'high' | 'xhigh' | 'max' nullable

Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, xhigh, and max. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. Setting to none disables reasoning entirely if the model supports.

prompt_cache_keystring nullable

A key to identify prompt cache for reuse across requests. If provided, the prompt will be cached and can be reused in subsequent requests with the same key.

cache_saltstring nullable

Isolates this request's prefix cache: only requests with the same salt reuse each other's cached prefix. Honoured by models served on vLLM; ignored elsewhere.

chat_template_kwargsobject nullable

Chat template kwargs.

continue_final_messageboolean nullable

If set, the final assistant message is used as a prefix for the model to continue generating from, rather than starting a new turn. Only applicable when the last message in the conversation is an assistant message.

ignore_eosboolean nullable

Keep generating until max_tokens instead of stopping at the end-of-sequence token. Only honoured on models tagged with the allow_ignore_eos feature flag; ignored otherwise. Intended for benchmarking, where a fixed output length is needed.

Example request

{
  "model": "meta-llama/Llama-2-70b-chat-hf"
}

Response

Successful Response

unknown required

Changes

  • 28e9378465caRevision changes
    • info

      added to the ///// request property anyOf list

  • b46a328dcee4Revision changes
    • info

      added the new optional request property