---
title: "Tokenize inputs"
method: POST
path: "/tokenize"
tags: ["Text Embeddings Inference"]
---

# Tokenize inputs

`POST /tokenize`

Tokenize inputs

## Request body

- TokenizeRequest
  - `add_special_tokens` boolean
  - `inputs` union, required
    - string
    - string[]

## Response `200`

Tokenized ids

- array[]
  - SimpleToken[]
    - `id` integer, required
    - `special` boolean, required
    - `start` integer, nullable
    - `stop` integer, nullable
    - `text` string, required

## Other responses

- `422` — Tokenization error

## Changes

- **2024-03-22** `36fa20faa0bb` — 2 breaking, 1 info
  - removed the required property `code` from the response with the `422` status
  - removed the required property `message` from the response with the `422` status
  - added the required property `error` to the response with the `422` status
- **2024-01-26** `51015074a126` — 1 info
  - endpoint added
- **2023-10-13** `8c00768ef703` — 1 breaking
  - api path removed without deprecation

[Change history](https://skmtc.dev/huggingface/apis/text-embeddings-inference/changes/tokenize/post.md)

---

[API](https://skmtc.dev/huggingface/apis/text-embeddings-inference.md) · [All operations](https://skmtc.dev/huggingface/apis/text-embeddings-inference/llms.txt) · [OpenAPI document](https://skmtc-service-production.skmtc.workers.dev/v1/apis/huggingface/text-embeddings-inference/revisions/36fa20faa0bb/schema)