ATatomaai
chat

Create chat completion

Changed on

This handler performs several key operations:

  1. Forwards the chat completion request to the inference service
  2. Signs the response using the node's keystore
  3. Tracks token usage for the stack

Arguments

  • Extension((stack_small_id, estimated_total_tokens)) - Stack ID and estimated tokens count from middleware
  • state - Application state containing the inference client and keystore
  • payload - The chat completion request body

Returns

Returns a JSON response containing:

  • The inference service's response
  • A cryptographic signature of the response

Errors

Returns a AtomaServiceError::InternalError if:

  • The inference service request fails
  • Response parsing fails
  • Response signing fails
  • Token usage update fails
post/v1/chat/completions

Request

Request body

frequency_penaltynumber float nullable

Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far

function_callunknown
logit_biasobject nullable

Modify the likelihood of specified tokens appearing in the completion.

Accepts a JSON object that maps tokens (specified by their token ID in the tokenizer) to an associated bias value from -100 to 100. Mathematically, the bias is added to the logits generated by the model prior to sampling. The exact effect will vary per model, but values between -1 and 1 should decrease or increase likelihood of selection; values like -100 or 100 should result in a ban or exclusive selection of the relevant token.

max_completion_tokensinteger nullable

The maximum number of tokens to generate in the chat completion

max_tokensinteger nullable

The maximum number of tokens to generate in the chat completion

modelstring required

ID of the model to use

ninteger nullable

How many chat completion choices to generate for each input message

parallel_tool_callsboolean nullable

Whether to enable parallel tool calls.

presence_penaltynumber float nullable

Number between -2.0 and 2.0. Positive values penalize new tokens based on whether they appear in the text so far

seedinteger nullable

If specified, our system will make a best effort to sample deterministically

service_tierstring nullable

Specifies the latency tier to use for processing the request. This parameter is relevant for customers subscribed to the scale tier service:

If set to 'auto', and the Project is Scale tier enabled, the system will utilize scale tier credits until they are exhausted. If set to 'auto', and the Project is not Scale tier enabled, the request will be processed using the default service tier with a lower uptime SLA and no latency guarantee. If set to 'default', the request will be processed using the default service tier with a lower uptime SLA and no latency guarantee. When not set, the default behavior is 'auto'.

stopstring[] nullable

Up to 4 sequences where the API will stop generating further tokens

streamboolean nullable

Whether to stream back partial progress

temperaturenumber float nullable

What sampling temperature to use, between 0 and 2

top_logprobsinteger nullable

An integer between 0 and 20 specifying the number of most likely tokens to return at each token position, each with an associated log probability. logprobs must be set to true if this parameter is used.

top_pnumber float nullable

An alternative to sampling with temperature

userstring nullable

A unique identifier representing your end-user

Example request

{
  "max_completion_tokens": 4096,
  "max_tokens": 4096,
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "n": 1,
  "seed": 123,
  "temperature": 0.7,
  "top_logprobs": 1,
  "top_p": 1,
  "user": "user-1234"
}

Response

Chat completion successful

createdinteger required

The Unix timestamp (in seconds) of when the chat completion was created.

idstring required

A unique identifier for the chat completion.

modelstring required

The model used for the chat completion.

objectstring required

The object of the chat completion.

service_tierstring nullable

The service tier of the chat completion.

system_fingerprintstring nullable

The system fingerprint for the completion, if applicable.

Changes

  • 9945200613eeRevision changes
    • breaking

      removed subschema #1 from the request property oneOf list

    • breaking

      removed subschema #1 subschema #2 subschema #3 subschema #4 from the / request property oneOf list

    • breaking

      added 'propertyNames' constraint to the request property ///

    • breaking

      the / request property type changed from string to integer, and format from no format to int32

    • breaking

      the request property format changed from no format to int32

    • breaking

      the request property type changed from no type to array null

    • breaking

      the /// request property type changed from no type to object null

    • breaking

      the response property // became nullable for the status

    • breaking

      the response property became nullable for the status

    • breaking

      the response property // became optional for the status

    • breaking

      the response property became optional for the status

    • breaking

      the response property became optional for the status

    • breaking

      added subschema #1 to the // response property oneOf list for the response status

    • breaking

      added subschema #1 to the response property oneOf list for the response status

    • breaking

      added subschema #1 subschema #2 subschema #3 subschema #4 to the // response property oneOf list for the response status

    • breaking

      the response's property type changed from object to no type for status

    • breaking

      removed the required property / from the response with the status

    • breaking

      removed the required property / from the response with the status

    • warning

      the / request property's min was set to 0.00

    • warning

      removed the request property

    • warning

      removed the optional property / from the response with the status

    • warning

      removed the optional property / from the response with the status

    • info

      added the new optional request property

    • info

      added the new optional request property

    • info

      added the new optional request property

    • info

      added the new optional request property

    • info

      added the new optional request property

    • info

      added the new optional request property

    • info

      added the new optional request property

    • info

      the request property became nullable

    • info

      the request property /// became nullable

    • info

      the request property default value [] was added

    • info

      added subschema #1 subschema #2 subschema #3 subschema #4 to the / request property oneOf list

    • info

      added the optional property to the response with the status

    • info

      removed the content_filter enum value from the // response property for the response status

    • info

      removed the length_capped enum value from the // response property for the response status

    • info

      removed the stopped enum value from the // response property for the response status

    • info

      removed subschema #1 subschema #2 subschema #3 subschema #4 from the // response property oneOf list for the response status

  • 1ae6677fa6b2Revision changes
    • info

      added the new optional request property

    • info

      request property deprecated