HUhuggingface
Text Embeddings Inference

Tokenize inputs

Changed on
post/tokenize

Request

  • The document declares no server URL.
  • Auth
    none declared

Request body

add_special_tokensboolean
prompt_namestring nullable

The name of the prompt that should be used by for encoding. If not set, no prompt will be applied.

Must be a key in the sentence-transformers configuration prompts dictionary.

For example if prompt_name is "query" and the prompts is {"query": "query: ", ...}, then the sentence "What is the capital of France?" will be encoded as "query: What is the capital of France?" because the prompt text will be prepended before any text to encode.

Example request

{
  "prompt_name": "null"
}

Response

Tokenized ids

Example response

[
  [
    {
      "id": 0,
      "special": false,
      "start": 0,
      "stop": 2,
      "text": "test"
    }
  ]
]

Changes

  • 9988d0e89620Revision changes
    • warning

      added the new Empty enum value to the response property for the response status

    • info

      added the non-success response with the status

    • info

      added the non-success response with the status

  • b923798e3868Revision changes
    • info

      added the new optional request property