FRfriendli
Serverless.Chat

Stream chat completions

Given a list of messages forming a conversation, the model generates a response.

post/serverless/v1/chat/completions#stream

Request

Headers

X-Friendli-Teamstring nullable

ID of team to run requests as (optional parameter).

Request body

modelstring required

Code of the model to use. See available model list.

chat_template_kwargsobject nullable

Additional keyword arguments supplied to the template renderer. These parameters will be available for use within the chat template.

eos_tokeninteger[] nullable

A list of endpoint sentence tokens.

frequency_penaltynumber nullable

Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled, taking into account their frequency in the preceding text. This penalization diminishes the model's tendency to reproduce identical lines verbatim.

logit_biasobject nullable

Accepts a JSON object that maps tokens to an associated bias value. Mathematically, the bias is added to the logits generated by the model prior to sampling. The exact effect will vary per model.

logprobsboolean nullable

Whether to return log probabilities of the output tokens or not.

max_tokensinteger nullable

The maximum number of tokens to generate. For decoder-only models like GPT, the length of your input tokens plus max_tokens should not exceed the model's maximum length (e.g., 2048 for OpenAI GPT-3). For encoder-decoder models like T5 or BlenderBot, max_tokens should not exceed the model's maximum output length. This is similar to Hugging Face's max_new_tokens argument.

min_pnumber nullable

A scaling factor used to determine the minimum token probability threshold. This threshold is calculated as min_p multiplied by the probability of the most likely token. Tokens with probabilities below this scaled threshold are excluded from sampling. Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values result in stricter filtering, while lower values allow for greater diversity. The default value of 0.0 disables filtering, allowing all tokens to be considered for sampling.

ninteger nullable

The number of independently generated results for the prompt. Defaults to 1. This is similar to Hugging Face's num_return_sequences argument.

parallel_tool_callsboolean nullable

Whether to enable parallel function calling.

presence_penaltynumber nullable

Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled at least once in the existing text.

repetition_penaltynumber nullable

Penalizes tokens that have already appeared in the generated result (plus the input tokens for decoder-only models). Should be positive value (1.0 means no penalty). See keskar et al., 2019 for more details. This is similar to Hugging Face's repetition_penalty argument.

reasoning_effort'low' | 'medium' | 'high' nullable

Determines the amount of reasoning effort the model applies when generating a response. Higher values may produce more detailed and thoughtful outputs, but can increase response time. This parameter is only effective for reasoning models.

reasoning_budgetinteger nullable

Specifies a positive integer that defines a limit on the number of tokens used for internal reasoning tokens. This parameter is only effective for reasoning models.

stopstring[] nullable

When one of the stop phrases appears in the generation result, the API will stop generation. The stop phrases are excluded from the result. Defaults to empty list.

streamboolean nullable

Whether to stream the generation result. When set to true, each token will be sent as server-sent events once generated.

parse_reasoningboolean nullable

Parses model reasoning into reasoning_content while keeping the answer in content. Default value may vary between endpoints.

For more detailed information, please refer here.

include_reasoningboolean nullable

When parse_reasoning=true, include parsed reasoning (reasoning_content). Defaults to true.

For more detailed information, please refer here.

temperaturenumber nullable

Sampling temperature. Smaller temperature makes the generation result closer to greedy, argmax (i.e., top_k = 1) sampling. Defaults to 1.0. This is similar to Hugging Face's temperature argument.

top_kinteger nullable

Limits sampling to the top k tokens with the highest probabilities. Values range from 0 (no filtering) to the model's vocabulary size (inclusive). The default value of 0 applies no filtering, allowing all tokens.

top_logprobsinteger nullable

The number of most likely tokens to return at each token position, each with an associated log probability. logprobs must be set to true if this parameter is used.

top_pnumber nullable

Keeps only the smallest set of tokens whose cumulative probabilities reach top_p or higher. Values range from 0.0 (exclusive) to 1.0 (inclusive). The default value of 1.0 includes all tokens, allowing maximum diversity.

xtc_thresholdnumber nullable

A probability threshold used to identify “top choice” tokens for exclusion in XTC (Exclude Top Choices) sampling. Tokens with probabilities at or above this threshold are considered viable candidates, and all but the least likely viable token are excluded from sampling. This option reduces the dominance of highly probable tokens while preserving some diversity by keeping the least confident “top choice.” Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values make the filtering more selective by requiring higher probabilities to trigger exclusion, while lower values apply filtering more broadly. The default value of 0.0 disables XTC filtering entirely.

xtc_probabilitynumber nullable

The probability that XTC (Exclude Top Choices) filtering will be applied for each sampling decision. When XTC is triggered, high-probability tokens above the xtc_threshold are excluded except for the least likely viable token. This stochastic activation allows for a balance between standard sampling and creativity-boosting exclusion filtering. Values range from 0.0 (inclusive) to 1.0 (inclusive), where 0.0 means XTC is never applied, 1.0 means XTC is always applied when viable tokens exist, and intermediate values provide probabilistic activation. The default value of 0.0 disables XTC filtering.

min_tokensinteger nullable

The minimum number of tokens to generate. Default value is 0. This is similar to Hugging Face's min_new_tokens argument.

This field is unsupported when tools or response_format is specified.

Example request

{
  "messages": [
    {
      "content": "You are a helpful assistant.",
      "role": "system"
    },
    {
      "content": "Hello!",
      "role": "user"
    }
  ],
  "model": "meta-llama/Llama-3.1-8B-Instruct"
}

Response

Successfully generated a chat response.

Changes

No recorded changes to this operation across all 1 revision of this API.