Sign in
Chat Completions

Openai Chat Completions

post/v1/chat/completions

Headers

x-deepinfra-sourcestring nullable
xi-api-keystring nullable
x-api-keystring nullable

Request body

service_tier'default' | 'priority' | 'flex'
fail_fastboolean

If true, the request is rejected immediately with HTTP 429 when the model has no spare capacity, instead of waiting in the queue. Opt-in; the default (false) keeps standard queueing behavior.

modelsstring[] nullable

Ordered list of up to 4 fallback models. The request is attempted on each model in order: when a model rejects it for lack of capacity (HTTP 429 model-busy / flex no-capacity), the next model is tried server-side. The first model that accepts serves the request; the response's model field and billing reflect that model, at that model's pricing. Models before the last are attempted without queueing (as if fail_fast were set); the last model honors the request's own fail_fast value. When models is set, the model field is ignored. Entries must be plain model names (no deploy_id:, custom_hostport, or :revision specifiers); duplicate entries are ignored, keeping the first occurrence.

modelstring required

model name

streamboolean

whether to stream the output via SSE or return the full response

temperaturenumber

What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic

top_pnumber

An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens comprising the top 10% probability mass are considered.

min_pnumber

Float that represents the minimum probability for a token to be considered, relative to the probability of the most likely token. Must be in [0, 1]. Set to 0 to disable this.

top_kinteger

Sample from the best k (number of) tokens. 0 means off

max_tokensinteger nullable

The maximum number of tokens to generate in the chat completion.

The total length of input tokens and generated tokens is limited by the model's context length. If explicitly set to None it will be the model's max context length minus input length or 65536, whichever is smaller.

stop_token_idsinteger[] nullable

Up to 16 token IDs where the API will stop generating further tokens. Merged with the model's built-in stop tokens. Intended for private deployments.

ninteger

number of sequences to return

presence_penaltynumber

Positive values penalize new tokens based on whether they appear in the text so far, increasing the model's likelihood to talk about new topics.

frequency_penaltynumber

Positive values penalize new tokens based on how many times they appear in the text so far, increasing the model's likelihood to talk about new topics.

repetition_penaltynumber

Alternative penalty for repetition, but multiplicative instead of additive (> 1 penalize, < 1 encourage)

userstring nullable

A unique identifier representing your end-user, which can help monitor and detect abuse. Avoid sending us any identifying information. We recommend hashing user identifiers.

seedinteger nullable

Seed for random number generator. If not provided, a random seed is used. Determinism is not guaranteed.

logprobsboolean nullable

Whether to return log probabilities of the output tokens or not.If true, returns the log probabilities of each output token returned in the content of message.

reasoning_effort'none' | 'minimal' | 'low' | 'medium' | 'high' | 'xhigh' | 'max' nullable

Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, xhigh, and max. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. Setting to none disables reasoning entirely if the model supports.

prompt_cache_keystring nullable

A key to identify prompt cache for reuse across requests. If provided, the prompt will be cached and can be reused in subsequent requests with the same key.

chat_template_kwargsobject nullable

Chat template kwargs.

continue_final_messageboolean nullable

If set, the final assistant message is used as a prefix for the model to continue generating from, rather than starting a new turn. Only applicable when the last message in the conversation is an assistant message.

ignore_eosboolean nullable

Keep generating until max_tokens instead of stopping at the end-of-sequence token. Only honoured on models tagged with the allow_ignore_eos feature flag; ignored otherwise. Intended for benchmarking, where a fixed output length is needed.

Example request

{
  "model": "meta-llama/Llama-2-70b-chat-hf"
}

Response

Successful Response

{"stackTrail":"paths:/v1/chat/completions:post:responses:200:content:application/json:schema","oasType":"schema","type":"unknown"}

Changes

Changed in 6 of the 17 revisions of this API.317

  • 900aef7c556d11See the full diff
    • ▲

      removed ChatCompletionToolMessage ChatCompletionAssistantMessage ChatCompletionUserMessage ChatCompletionSystemMessage from the messages/items/ request property anyOf list

      request-property-any-of-removed

    • ○

      added ChatCompletionToolMessage ChatCompletionAssistantMessage ChatCompletionUserMessage ChatCompletionSystemMessage to the messages/items/ request property oneOf list

      request-property-one-of-added

  • 63ee8b82144f1See the full diff
    • ○

      added the new optional request property models

      new-optional-request-property

  • 2df28210837a13See the full diff
    • ▲

      request property tool_choice/anyOf[subschema #1]/ was restricted to a list of enum values

      request-property-became-enum

    • ○

      added the new auto enum value to the request property tool_choice/anyOf[subschema #1]/

      request-property-enum-value-added

    • ○

      added the new none enum value to the request property tool_choice/anyOf[subschema #1]/

      request-property-enum-value-added

    • ○

      added the new required enum value to the request property tool_choice/anyOf[subschema #1]/

      request-property-enum-value-added

  • 2418bb7a26911See the full diff
    • ○

      added the new optional request property ignore_eos

      new-optional-request-property

  • ba92d784991412See the full diff
    • ▲

      removed ChatTools from the tool_choice request property anyOf list

      request-property-any-of-removed

    • ○

      added the new optional request property fail_fast

      new-optional-request-property

    • ○

      added FunctionTool to the tool_choice request property anyOf list

      request-property-any-of-added

    This revision also has 3 changes that name no endpoint, such as unreferenced schemas being removed. See the revision's changelog

  • 09df98e6ea159See the full diff
    • ○

      added the new optional request property messages/items/anyOf[subschema #1: ChatCompletionToolMessage]/content/anyOf[subschema #2]/items/prompt_cache_breakpoint

      new-optional-request-property

    • ○

      added the new optional request property messages/items/anyOf[subschema #2: ChatCompletionAssistantMessage]/content/anyOf[subschema #2]/items/prompt_cache_breakpoint

      new-optional-request-property

    • ○

      added the new optional request property messages/items/anyOf[subschema #3: ChatCompletionUserMessage]/content/anyOf[subschema #2]/items/anyOf[subschema #1: ChatCompletionContentPartText]/prompt_cache_breakpoint

      new-optional-request-property

    • ○

      added the new optional request property messages/items/anyOf[subschema #4: ChatCompletionSystemMessage]/content/anyOf[subschema #2]/items/prompt_cache_breakpoint

      new-optional-request-property

    • ○

      added the new optional request property prompt_cache_options

      new-optional-request-property

    • ○

      added the new max enum value to the request property reasoning/anyOf[subschema #1: ChatReasoningSettings]/effort/anyOf[subschema #1]/

      request-property-enum-value-added

    • ○

      added the new max enum value to the request property reasoning_effort/anyOf[subschema #1]/

      request-property-enum-value-added

    • ○

      added the new minimal enum value to the request property reasoning/anyOf[subschema #1: ChatReasoningSettings]/effort/anyOf[subschema #1]/

      request-property-enum-value-added

    • ○

      added the new minimal enum value to the request property reasoning_effort/anyOf[subschema #1]/

      request-property-enum-value-added