Tokenize

This endpoint splits input text into smaller units called tokens using byte-pair encoding (BPE). To learn more about tokenization and byte pair encoding, see the tokens page.

post/v1/tokenize

Request

  • Base URL: https://api.cohere.com
  • URL: https://api.cohere.com/v1/tokenize
  • Auth: HTTP bearer

Headers

X-Client-Namestring

The name of the project that is making the request.

Request body

textstring required

The string to be tokenized, the minimum text length is 1 character, and the maximum text length is 65536 characters.

modelstring required

The input will be tokenized by the tokenizer that is used by this model.

Response

OK

tokensinteger[] required

An array of tokens, where each token is an integer.

token_stringsstring[] required

Changes

No recorded changes to this operation across all 1 revision of this API.