---
title: "Tokenize"
method: POST
path: "/v1/tokenize"
---

# Tokenize

`POST /v1/tokenize`

## Request

- Base URL: `https://api.cohere.com`
- URL: `https://api.cohere.com/v1/tokenize`
- Auth: HTTP bearer

This endpoint splits input text into smaller units called tokens using byte-pair encoding (BPE). To learn more about tokenization and byte pair encoding, see the tokens page.

## Headers

- `X-Client-Name` string

## Request body

- object
  - `text` string, required: The string to be tokenized, the minimum text length is 1 character, and the maximum text length is 65536 characters.
  - `model` string, required: The input will be tokenized by the tokenizer that is used by this model.

## Response `200`

OK

- TokenizeResponse200
  - `tokens` integer[], required: An array of tokens, where each token is an integer.
  - `token_strings` string[], required
  - `meta` ApiMeta
    - `api_version` ApiMetaApiVersion
      - `version` string, required
      - `is_deprecated` boolean
      - `is_experimental` boolean
    - `billed_units` ApiMetaBilledUnits
      - `images` number, double: The number of billed images.
      - `input_tokens` number, double: The number of billed input tokens.
      - `image_tokens` number, double: The number of billed image tokens.
      - `output_tokens` number, double: The number of billed output tokens.
      - `search_units` number, double: The number of billed search units.
      - `classifications` number, double: The number of billed classifications units.
      - `pages` number, double: The number of billed pages parsed.
    - `tokens` ApiMetaTokens
      - `input_tokens` number, double: The number of tokens used as input to the model.
      - `output_tokens` number, double: The number of tokens produced by the model.
    - `cached_tokens` number, double: The number of prompt tokens that hit the inference cache.
    - `warnings` string[]

## Other responses

- `400`: This error is returned when the request is not well formed. This could be because: - JSON is invalid - The request is missing required fields - The request contains an invalid combination of fields
- `401`: This error indicates that the operation attempted to be performed is not allowed. This could be because: - The api token is invalid - The user does not have the necessary permissions
- `403`: This error indicates that the operation attempted to be performed is not allowed. This could be because: - The api token is invalid - The user does not have the necessary permissions
- `404`: This error is returned when a resource is not found. This could be because: - The endpoint does not exist - The resource does not exist eg model id, dataset id
- `422`: This error is returned when the request is not well formed. This could be because: - JSON is invalid - The request is missing required fields - The request contains an invalid combination of fields
- `429`: Too many requests
- `498`: This error is returned when a request or response contains a deny-listed token.
- `499`: This error is returned when a request is cancelled by the user.
- `500`: This error is returned when an uncategorised internal server error occurs.
- `501`: This error is returned when the requested feature is not implemented.
- `503`: This error is returned when the service is unavailable. This could be due to: - Too many users trying to access the service at the same time
- `504`: This error is returned when a request to the server times out. This could be due to: - An internal services taking too long to respond

---

[API](https://skmtc.dev/cohere-ai/apis/cohere-api.md) · [All operations](https://skmtc.dev/cohere-ai/apis/cohere-api/llms.txt) · [OpenAPI document](https://skmtc.dev/cohere-ai/apis/cohere-api/revisions/8e538a293bd3?raw)