tokenize

Tokenize text

Tokenizes the provided text using the specified tokenization method. This is a stateless utility endpoint useful for debugging and understanding how text will be processed during indexing and querying. The response includes both the indexed tokens (as stored in the inverted index) and query tokens (after optional stopword removal).

post/tokenize

Request

  • Base URL: https://localhost:8080/v1
  • URL: https://localhost:8080/v1/tokenize
  • Auth: one of:
    • none
    • OAuth 2 (implicit, authorization URL http://to-be-configured-in-the-application-config)

Request body

textstring required

The text to tokenize.

tokenization'word' | 'lowercase' | 'whitespace' | 'field' | 'trigram' | 'gse' | 'kagome_kr' | 'kagome_ja' | 'gse_ch' required

The tokenization method to apply.

stopwordPresetsobject

Optional user-defined named stopword presets. Shape matches InvertedIndexConfig.stopwordPresets on a collection: each key is a preset name, each value is a plain list of stopwords. A preset name that matches a built-in ('en', 'none') fully replaces the built-in. Preset names must not be empty or whitespace-only; each word list must contain at least one word; individual words must not be empty or whitespace-only. Mutually exclusive with stopwords — pass one or the other, not both.

Response

Successfully tokenized the text.

indexedstring[]

The tokens as they would be stored in the inverted index.

querystring[]

The tokens as they would be used for query matching (e.g., after stopword removal).

Changes

No recorded changes to this operation across all 1 revision of this API.