ATatomaai
Chat

Create chat completions

Changed on

This function processes chat completion requests by determining whether to use streaming or non-streaming response handling based on the request payload. For streaming requests, it configures additional options to track token usage.

Returns

Returns a Response containing either:

  • A streaming SSE connection for real-time completions
  • A single JSON response for non-streaming completions

Errors

Returns an error status code if:

  • The request processing fails
  • The streaming/non-streaming handlers encounter errors
  • The underlying inference service returns an error
post/v1/chat/completions

Request

Request body

frequency_penaltynumber float nullable

Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far

function_callunknown
logit_biasobject nullable

Modify the likelihood of specified tokens appearing in the completion.

Accepts a JSON object that maps tokens (specified by their token ID in the tokenizer) to an associated bias value from -100 to 100. Mathematically, the bias is added to the logits generated by the model prior to sampling. The exact effect will vary per model, but values between -1 and 1 should decrease or increase likelihood of selection; values like -100 or 100 should result in a ban or exclusive selection of the relevant token.

max_completion_tokensinteger nullable

The maximum number of tokens to generate in the chat completion

max_tokensinteger nullable

The maximum number of tokens to generate in the chat completion

modelstring required

ID of the model to use

ninteger nullable

How many chat completion choices to generate for each input message

parallel_tool_callsboolean nullable

Whether to enable parallel tool calls.

presence_penaltynumber float nullable

Number between -2.0 and 2.0. Positive values penalize new tokens based on whether they appear in the text so far

seedinteger nullable

If specified, our system will make a best effort to sample deterministically

service_tierstring nullable

Specifies the latency tier to use for processing the request. This parameter is relevant for customers subscribed to the scale tier service:

If set to 'auto', and the Project is Scale tier enabled, the system will utilize scale tier credits until they are exhausted. If set to 'auto', and the Project is not Scale tier enabled, the request will be processed using the default service tier with a lower uptime SLA and no latency guarantee. If set to 'default', the request will be processed using the default service tier with a lower uptime SLA and no latency guarantee. When not set, the default behavior is 'auto'.

stopstring[] nullable

Up to 4 sequences where the API will stop generating further tokens

streamboolean nullable

Whether to stream back partial progress. Must be false for this request type.

temperaturenumber float nullable

What sampling temperature to use, between 0 and 2

top_logprobsinteger nullable

An integer between 0 and 20 specifying the number of most likely tokens to return at each token position, each with an associated log probability. logprobs must be set to true if this parameter is used.

top_pnumber float nullable

An alternative to sampling with temperature

userstring nullable

A unique identifier representing your end-user

Example request

{
  "functions": [
    {
      "name": "get_current_weather",
      "description": "Get the current weather in a location",
      "parameters": {
        "type": "object",
        "properties": {
          "location": {
            "type": "string",
            "description": "The location to get the weather for"
          }
        },
        "required": [
          "location"
        ]
      }
    }
  ],
  "logit_bias": {
    "1234567890": 0.5,
    "1234567891": -0.5
  },
  "max_completion_tokens": 4096,
  "max_tokens": 4096,
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful AI assistant"
    },
    {
      "role": "user",
      "content": "Hello!"
    },
    {
      "role": "assistant",
      "content": "I'm here to help you with any questions you have. How can I assist you today?"
    }
  ],
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "n": 1,
  "parallel_tool_calls": true,
  "seed": 123,
  "service_tier": "auto",
  "temperature": 0.7,
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_current_weather",
        "description": "Get the current weather in a location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {
              "type": "string",
              "description": "The location to get the weather for"
            }
          },
          "required": [
            "location"
          ]
        }
      }
    }
  ],
  "top_logprobs": 1,
  "top_p": 1,
  "user": "user-1234"
}

Response

Chat completions

createdinteger required

The Unix timestamp (in seconds) of when the chat completion was created.

idstring required

A unique identifier for the chat completion.

modelstring required

The model used for the chat completion.

objectstring required

The object of the chat completion.

service_tierstring nullable

The service tier of the chat completion.

system_fingerprintstring nullable

The system fingerprint for the completion, if applicable.

Changes

  • f0bf5a2dde0fRevision changes
    • info

      added the required property ////////// to the response with the status

  • 972a6a82c154Revision changes
    • breaking

      removed subschema #1 subschema #2 subschema #3 subschema #4 from the ///// request property oneOf list

    • breaking

      added subschema #1: System subschema #2: User subschema #3: Assistant subschema #4: Tool to the // response property oneOf list for the response status

    • info

      added subschema #1: System subschema #2: User subschema #3: Assistant subschema #4: Tool to the ///// request property oneOf list

    • info

      removed subschema #1 subschema #2 subschema #3 subschema #4 from the // response property oneOf list for the response status