Sign in
inference-proxy-api-new_Chat

Creates a model response for the given chat conversation.

post/chat/completions

Request body

metadataMetadata nullable

Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

temperaturenumber nullable

What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic. We generally recommend altering this or top_p but not both.

top_pnumber nullable

An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens comprising the top 10% probability mass are considered.

We generally recommend altering this or temperature but not both.

userstring

A unique identifier representing your end-user, which can help DigitalOcean to monitor and detect abuse.

modelstring required

Model ID used to generate the response.

max_tokensinteger nullable

The maximum number of tokens that can be generated in the completion.

The token count of your prompt plus max_tokens cannot exceed the model's context length.

max_completion_tokensinteger nullable

The maximum number of completion tokens that may be used over the course of the run. The run will make a best effort to use only the number of completion tokens specified, across multiple turns of the run.

frequency_penaltynumber nullable

Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model's likelihood to repeat the same line verbatim.

presence_penaltynumber nullable

Number between -2.0 and 2.0. Positive values penalize new tokens based on whether they appear in the text so far, increasing the model's likelihood to talk about new topics.

top_logprobsinteger nullable

An integer between 0 and 20 specifying the number of most likely tokens to return at each token position, each with an associated log probability. logprobs must be set to true if this parameter is used.

streamboolean nullable

If set to true, the model response data will be streamed to the client as it is generated using server-sent events.

logit_biasobject nullable

Modify the likelihood of specified tokens appearing in the completion.

Accepts a JSON object that maps tokens (specified by their token ID in the tokenizer) to an associated bias value from -100 to 100. Mathematically, the bias is added to the logits generated by the model prior to sampling. The exact effect will vary per model, but values between -1 and 1 should decrease or increase likelihood of selection; values like -100 or 100 should result in a ban or exclusive selection of the relevant token.

logprobsboolean nullable

Whether to return log probabilities of the output tokens or not. If true, returns the log probabilities of each output token returned in the content of message.

ninteger nullable

How many chat completion choices to generate for each input message. Note that you will be charged based on the number of generated tokens across all of the choices. Keep n as 1 to minimize costs.

Example request

{
  "temperature": 1,
  "top_p": 1,
  "user": "user-1234",
  "model": "llama3-8b-instruct",
  "stop": "\n",
  "n": 1
}

Response

OK

idstring required

A unique identifier for the chat completion.

createdinteger required

The Unix timestamp (in seconds) of when the chat completion was created.

modelstring required

The model used for the chat completion.

object'chat.completion' required

The object type, which is always chat.completion.

Changes

Changed in 4 of the 8 revisions of this API.51127

  • cb3bf9b214594210See the full diff
    • ▲

      the ////// request property type changed from string to no type

      request-property-type-changed

    • ▲

      the ////// request property type changed from string to no type

      request-property-type-changed

    • ▲

      the ////// request property type changed from string to no type

      request-property-type-changed

    • ▲

      the ////// request property type changed from string to no type

      request-property-type-changed

    • ●

      added the new content_filter enum value to the // response property for the response status (media type: application/json)

      response-property-enum-value-added

    • ●

      added the new content_filter enum value to the // response property for the response status (media type: text/event-stream)

      response-property-enum-value-added

    • ○

      the endpoint scheme security ModelAccessKey was added to the API

      api-security-added

    • ○

      the endpoint scheme security ApiKeyAuth was removed from the API

      api-security-removed

    • ○

      api tag inference-proxy-api-new_Chat added

      api-tag-added

    • ○

      api tag Chat removed

      api-tag-removed

    • ○

      added subschema #1 subschema #2 to the ////// request property oneOf list

      request-property-one-of-added

    • ○

      added subschema #1 subschema #2 to the ////// request property oneOf list

      request-property-one-of-added

    • ○

      added subschema #1 subschema #2 to the ////// request property oneOf list

      request-property-one-of-added

    • ○

      added subschema #1 subschema #2 to the ////// request property oneOf list

      request-property-one-of-added

    • ○

      added the optional property /// to the response with the status (media type: text/event-stream)

      response-optional-property-added

    • ○

      added the required property /// to the response with the status (media type: application/json)

      response-required-property-added

    This revision also has 3 changes that name no endpoint, such as unreferenced schemas being removed. See the revision's changelog

  • f8e8c290636c27See the full diff
    • ●

      added the new tool_calls enum value to the // response property for the response status (media type: application/json)

      response-property-enum-value-added

    • ●

      added the new tool_calls enum value to the // response property for the response status (media type: text/event-stream)

      response-property-enum-value-added

    • ○

      added the new optional request property ////

      new-optional-request-property

    • ○

      added the new optional request property /

      new-optional-request-property

    • ○

      added the new optional request property /

      new-optional-request-property

    • ○

      added tool discriminator mapping keys to the // request property

      request-property-discriminator-mapping-added

    • ○

      added to the // request property oneOf list

      request-property-one-of-added

    • ○

      added the optional property /// to the response with the status (media type: text/event-stream)

      response-optional-property-added

    • ○

      added the optional property /// to the response with the status (media type: application/json)

      response-optional-property-added

    This revision also has 70 changes that name no endpoint, such as unreferenced schemas being removed. See the revision's changelog

  • 5fc9af728e82153See the full diff
    • ▲

      removed from the // request property oneOf list

      request-property-one-of-removed

    • ●

      removed the request property ////

      request-property-removed

    • ●

      removed the request property /

      request-property-removed

    • ●

      removed the request property /

      request-property-removed

    • ●

      removed the optional property /// from the response with the status (media type: text/event-stream)

      response-optional-property-removed

    • ●

      removed the optional property /// from the response with the status (media type: application/json)

      response-optional-property-removed

    • ○

      removed tool discriminator mapping keys from the // request property

      request-property-discriminator-mapping-deleted

    • ○

      removed the tool_calls enum value from the // response property for the response status (media type: application/json)

      response-property-enum-value-removed

    • ○

      removed the tool_calls enum value from the // response property for the response status (media type: text/event-stream)

      response-property-enum-value-removed

    This revision also has 436 changes that name no endpoint, such as unreferenced schemas being removed. See the revision's changelog

  • f8e8c290636c27See the full diff
    • ●

      added the new tool_calls enum value to the // response property for the response status (media type: application/json)

      response-property-enum-value-added

    • ●

      added the new tool_calls enum value to the // response property for the response status (media type: text/event-stream)

      response-property-enum-value-added

    • ○

      added the new optional request property ////

      new-optional-request-property

    • ○

      added the new optional request property /

      new-optional-request-property

    • ○

      added the new optional request property /

      new-optional-request-property

    • ○

      added tool discriminator mapping keys to the // request property

      request-property-discriminator-mapping-added

    • ○

      added to the // request property oneOf list

      request-property-one-of-added

    • ○

      added the optional property /// to the response with the status (media type: text/event-stream)

      response-optional-property-added

    • ○

      added the optional property /// to the response with the status (media type: application/json)

      response-optional-property-added

    This revision also has 70 changes that name no endpoint, such as unreferenced schemas being removed. See the revision's changelog