---
title: "Tokenize inputs"
method: POST
path: "/tokenize"
tags: ["Text Embeddings Inference"]
---

# Tokenize inputs

`POST /tokenize`

## Request body

- TokenizeRequest
  - `add_special_tokens` boolean
  - `inputs` union, required
    - string
    - string[]
  - `prompt_name` string, nullable — The name of the prompt that should be used by for encoding. If not set, no prompt will be applied. Must be a key in the `sentence-transformers` configuration `prompts` dictionary. For example if ``prompt_name`` is "query" and the ``prompts`` is {"query": "query: ", ...}, then the sentence "What is the capital of France?" will be encoded as "query: What is the capital of France?" because the prompt text will be prepended before any text to encode.

## Response `200`

Tokenized ids

- array[]
  - SimpleToken[]
    - `id` integer, required
    - `special` boolean, required
    - `start` integer, nullable
    - `stop` integer, nullable
    - `text` string, required

## Other responses

- `400` — Batch is empty
- `413` — Batch size error
- `422` — Tokenization error

## Changes

- **2025-09-01** `9988d0e89620` — 1 warning, 2 info
  - added the new `Empty` enum value to the `error_type` response property for the response status `422`
  - added the non-success response with the status `400`
  - added the non-success response with the status `413`
- **2024-06-28** `b923798e3868` — 1 info
  - added the new optional request property `prompt_name`
- **2024-03-22** `36fa20faa0bb` — 2 breaking, 1 info
  - removed the required property `code` from the response with the `422` status
  - removed the required property `message` from the response with the `422` status
  - added the required property `error` to the response with the `422` status
- **2024-01-26** `51015074a126` — 1 info
  - endpoint added
- **2023-10-13** `8c00768ef703` — 1 breaking
  - api path removed without deprecation

[Change history](https://skmtc.dev/huggingface/apis/text-embeddings-inference/changes/tokenize/post.md)

---

[API](https://skmtc.dev/huggingface/apis/text-embeddings-inference.md) · [All operations](https://skmtc.dev/huggingface/apis/text-embeddings-inference/llms.txt) · [OpenAPI document](https://skmtc-service-production.skmtc.workers.dev/v1/apis/huggingface/text-embeddings-inference/revisions/f4c15add6d2b/schema)