OCoctoml
text

Create Completion

Changed on

Create a Completion.

post/v1/completions

Request

Request body

best_ofinteger

Number of sequences that are generated from the prompt.best_of must be greater than or equal to n.

echoboolean

Echo back the prompt in addition to the completion.

frequency_penaltynumber

Penalizes new tokens based on their frequency in the generated text so far.

ignore_eosboolean

Whether to ignore the EOS token and continue generating tokens after the EOS token is generated.

log_promptboolean

OctoAI stores request prompt if True.

logit_biasobject nullable

Modify the likelihood of specified tokens appearing in the completion. Accepts a JSON object that maps tokens (specified by their token ID in the tokenizer) to an associated bias value from -100 to 100. Mathematically, the bias is added to the logits generated by the model prior to sampling. The exact effect will vary per model, but values between -1 and 1 should decrease or increase likelihood of selection; values like -100 or 100 should result in a ban or exclusive selection of the relevant token. As an example, you can pass {'50256': -100} to prevent the <|endoftext|> token from being generated.

loglikelihoodboolean

Return log probabilities for all prompt tokens excluding the first one from prefill step if True.

logprobsinteger nullable

Number of log probabilities to return per output token.

max_tokensinteger

Maximum number of tokens to generate per output sequence.

min_pnumber

Sets a dynamic threshold of the top tokens to consider based on the probability value of the most likely next token. Following https://arxiv.org/abs/2407.01082.

modelstring required

The identifier of the model to use. Can be a shared tenancy or custom model identifier.

ninteger

Number of output sequences to return.

peftstring nullable

Parameter-efficient fine-tuning ID.

presence_penaltynumber

Penalizes new tokens based on whether they appear in the generated text so far.

repetition_penaltynumber

Controls the likelihood of the model generating repeated texts.

seedinteger

If specified, our system will make a best effort to sample deterministically, such that repeated requests with the same seed and parameters should return the same result. Determinism is not guaranteed, and you should refer to the system_fingerprint response parameter to monitor changes in the backend.

streamboolean

If set, tokens will be streamed incrementally to users. One or more tokens will be sent as server-sent events (SSE) as they become available. For more information on read above.

suffixstring nullable

The suffix that comes after a completion of inserted text.

temperaturenumber

Controls the randomness of the sampling.

top_pnumber

Controls the cumulative probability of the top tokens to consider.

userstring nullable

A unique identifier representing your end-user.

Example request

{
  "peft": "example_peft_id"
}

Response

An OctoAI text endpoint can be requested in either a synchronous orstreaming mode.When the request body has stream: False set the content type will beapplication/json. When the request body has stream: True set thecontent type will be text-event-stream and will respond with a stream ofserver-send-events (SSE).

createdinteger required

The Unix timestamp (in seconds) of when the completion was created.

idstring required

A unique identifier for the completion.

modelstring required

The model used for the completion.

object'text_completion'

Object type field which is a constant and preset.

system_fingerprintstring required

The system response fingerprint.

Example response

{
  "choices": [],
  "created": 11,
  "id": "cmpl-123",
  "model": "my_model",
  "system_fingerprint": "system_fingerprint"
}

Changes

  • 790b47aba5c5Revision changes
    • breaking

      added to the request property allOf list

    • breaking

      removed subschema #2 from the request property anyOf list

    • breaking

      the response's property type changed from no type to string for status (media type: application/json)

    • breaking

      the response's property type changed from no type to string for status

    • warning

      added the new error enum value to the response property for the response status

    • warning

      added the new text_completion enum value to the response property for the response status (media type: application/json)

    • warning

      added the new text_completion enum value to the response property for the response status (media type: text/event-stream)

    • info

      added the new optional request property

    • info

      added the new optional request property

    • info

      the request property default value changed from 16 to 512

  • 370f4057658eRevision changes
    • info

      added the new optional request property