LLM API

Tokenize text with a supported Algenta tokenizer model

Runs the selected tokenizer model and returns the produced tokens. This initial LLM API tranche currently exposes deterministic word tokenization.

post/v1/tokenize

Request

  • Base URL: https://api.algenta.ai
  • URL: https://api.algenta.ai/v1/tokenize
  • Auth: one of:
    • HTTP bearer
    • API key in header X-API-Key

Request body

modelstring

Model identifier from GET /v1/models.

inputstring required

UTF-8 text to tokenize.

Response

Successful Response

object'tokenization'
modelstring required
tokenizer_kindstring required
tokensstring[] required
token_countinteger required

Changes

No recorded changes to this operation across all 1 revision of this API.