---
title: "Tokenization by MODEL_ID"
method: POST
path: "/ai/tokenization/{MODEL_ID}"
tags: ["Get tokens"]
---

# Tokenization by MODEL_ID

`POST /ai/tokenization/{MODEL_ID}`

The tokenization request for the pre-trained and custom embedding use cases and specified embedding `modelId` (model name) sends text to return results in formats supported by embedding models.

## Headers

- `Authorization: Bearer ACCESS_TOKEN` string
- `Content-Type` string

## Request body

- TokenizationRequest
  - `batch` object[] — The text used as the input for the request.
    - `text` string — The content sent in the request.
  - `useCaseConfig` UseCaseConfig
    - `dataType` string — This optional parameter enables model-specific handling in the Prediction API to help improve model accuracy. Use the most applicable fields based on available dataTypes and the dataType value that best aligns with the text sent to the Prediction API. The two string values to use for embedding models are: * `"dataType": "query"` for the query. For query-to-query pairing, best practice is to use `dataType=query` on both API calls. * `"dataType": "passage"` for fields searched at query time. For example, if questions and answers from a FAQ are indexed, the value for questions is `"dataType": "query"` and the value for the answers is `"dataType": "passage"`.
  - `modelConfig` ModelConfig — Provides fields and values that specify ranges for tokens.
    - `vectorQuantizationMethod` string — Vector quantization compresses data size, as well as reducing memory usage. The methods are: * `min-max` - Creates tensors of the text and converts it to uint8 by normalizing it to the range [0, 255]. * `max-scale` - Finds the maximum absolute value for the encoded text, normalizes it by scaling the text to a range of -127 to 127, and then returns the quantized text as an 8-bit integer tensor.
    - `dimReductionSize` integer — Used to reduce vector size while maintaining good quality. This field allows any integer above 0, but less than or equal to the vector dimension of the model. If you send a vector dimension larger than the model, a 400 Bad Request error is returned. Not every model is designed to support this parameter. In this scenario, a warning message is generated that indicates quality can decrease.

## Response `200`

OK

- TokenizationResponse
  - `generatedTokens` object[]
    - `tokens` string[] — The array of tokens derived from the text submitted in the request. For example: "generatedTokens": [ { "tokens": [ "[CLS]", "query", ":", "mr", ".", "and", "mrs", ".", "du", "##rs", "##ley", "and", "o", "'", "malley", ",", "of", "number", "four", ",", "pri", "##vet", "drive", ",", "were", "proud", "to", "say", "that", "they", "were", "perfectly", "normal", ",", "thank", "you", "very", "much", ".", "[SEP]" ],
  - `tokensUsed` Token
    - `promptTokens` integer — The number of tokens generated to prompt the model to continue generating results.
    - `completionTokens` integer — The number of tokens used until the model completes.
    - `totalTokens` integer — The sum of the prompt and completion tokens used in the model.

---

[API](https://skmtc.dev/lucidworks/apis/rules-rewrites-api.md) · [All operations](https://skmtc.dev/lucidworks/apis/rules-rewrites-api/llms.txt) · [OpenAPI document](https://skmtc-service-production.skmtc.workers.dev/v1/apis/lucidworks/rules-rewrites-api/revisions/f2d3747848e8/schema)