Token Chunker

Splits text into chunks based on token count, ensuring each chunk stays within specified token limits.

post/v1/chunk/token

Response

Successful Response: A list of Chunk objects.

textstring

The actual text content of the chunk.

start_indexinteger

The starting character index of the chunk within the original input text.

end_indexinteger

The ending character index (exclusive) of the chunk within the original input text.

token_countinteger

The number of tokens in this specific chunk, according to the tokenizer used.

contextstring nullable

Chunk context.

Changes

Changed in 10 of the 20 revisions of this API.9619

May 4, 202533dc37e4abcd1See the full diff
- ○
  added the optional property items/context to the response with the 200 status
  response-optional-property-added
Apr 27, 2025bc64e17759521See the full diff
- ○
  endpoint added
  endpoint-added
This revision also has 1 change that name no endpoint, such as unreferenced schemas being removed. See the revision's changelog
Apr 27, 20255c8e448048061See the full diff
- ▲
  api removed without deprecation
  api-removed-without-deprecation
This revision also has 19 changes that name no endpoint, such as unreferenced schemas being removed. See the revision's changelog
Apr 27, 20257e3cb743a9ed1See the full diff
- ●
  removed the request property text
  request-property-removed
Apr 22, 2025b5fe83cf70c611See the full diff
- ▲
  removed the media type application/json from the request body
  request-body-media-type-removed
- ○
  added the media type multipart/form-data to the request body
  request-body-media-type-added
Apr 6, 2025502b267e607d12See the full diff
- ▲
  request property return_type was restricted to a list of enum values
  request-property-became-enum
- ○
  added the new chunks enum value to the request property return_type
  request-property-enum-value-added
- ○
  added the new texts enum value to the request property return_type
  request-property-enum-value-added
Apr 6, 20252d174e4e5a5a4See the full diff
- ○
  the request property chunk_overlap became optional
  request-property-became-optional
- ○
  the request property chunk_size became optional
  request-property-became-optional
- ○
  the request property return_type became optional
  request-property-became-optional
- ○
  the request property tokenizer became optional
  request-property-became-optional
Apr 6, 2025a9d5b6b90675414See the full diff
- ▲
  the response property items/end_index became optional for the status 200
  response-property-became-optional
- ▲
  the response property items/start_index became optional for the status 200
  response-property-became-optional
- ▲
  the response property items/text became optional for the status 200
  response-property-became-optional
- ▲
  the response property items/token_count became optional for the status 200
  response-property-became-optional
- ●
  removed the request property args
  request-property-removed
- ○
  added the new required request property chunk_overlap with a default value
  new-required-request-property-with-default
- ○
  added the new required request property chunk_size with a default value
  new-required-request-property-with-default
- ○
  added the new required request property return_type with a default value
  new-required-request-property-with-default
- ○
  added the new required request property tokenizer with a default value
  new-required-request-property-with-default
Apr 6, 2025678e906303363See the full diff
- ○
  removed the non-success response with the status 401
  response-non-success-status-removed
- ○
  removed the non-success response with the status 422
  response-non-success-status-removed
- ○
  removed the non-success response with the status 500
  response-non-success-status-removed
Apr 6, 2025c791be50601b243See the full diff
- ▲
  added the new required request property args
  new-required-request-property
- ▲
  the response's body type/format changed from / to array/ for status 200
  response-body-type-changed
- ●
  removed the request property chunk_overlap
  request-property-removed
- ●
  removed the request property chunk_size
  request-property-removed
- ●
  removed the request property return_type
  request-property-removed
- ●
  removed the request property tokenizer
  request-property-removed
- ○
  removed ErrorRES subschema #1 subschema #2 subschema #3 subschema #4 from the response body anyOf list for the response status 200
  response-body-any-of-removed
- ○
  added the non-success response with the status 401
  response-non-success-status-added
- ○
  added the non-success response with the status 500
  response-non-success-status-added

Full history