---
title: "Tokenize input"
method: POST
path: "/tokens"
tags: ["tokens"]
---

# Tokenize input

`POST /tokens`

## Request

- Base URL: `https://api.rosette.com/rest/v1`
- URL: `https://api.rosette.com/rest/v1/tokens`
- Auth: API key in header `X-RosetteAPI-Key`

Divides the input into tokens. <br><br>The response contains a list of tokens.

## Request body

- union
  - object
    - `content` string
    - `language` string: 3 letter code of language.
    - `options` object
      - `modelType` 'default' | 'perceptron' | 'DNN': Model type to use for Chinese, Japanese, Korean, or Thai. For Korean input without spaces, use `DNN`. For Chinese, Japanese, and Thai, use `perceptron` for an alternative algorithm
      - `disambiguatorType` 'perceptron' | 'DNN' | 'dictionary': For Hebrew only, determines disambiguator used. Default is `perceptron`. `DNN` is a deep nueral network model.
  - object
    - `contentUri` string
    - `language` string: 3 letter code of language.
    - `options` object
      - `modelType` 'default' | 'perceptron' | 'DNN': Model type to use for Chinese, Japanese, Korean, or Thai. For Korean input without spaces, use `DNN`. For Chinese, Japanese, and Thai, use `perceptron` for an alternative algorithm
      - `disambiguatorType` 'perceptron' | 'DNN' | 'dictionary': For Hebrew only, determines disambiguator used. Default is `perceptron`. `DNN` is a deep nueral network model.

## Response `200`

OK

- TokensResponse
  - `tokens` string[]: list of tokens

---

[API](https://skmtc.dev/babelstreet/apis/rosette-text-analytics.md) · [All operations](https://skmtc.dev/babelstreet/apis/rosette-text-analytics/llms.txt) · [OpenAPI document](https://skmtc.dev/babelstreet/apis/rosette-text-analytics/revisions/7c1d1497a5dd?raw)