Models

Advisory memory verdicts for several models

Advisory verdicts for up to 32 models against ONE host budget reading, so every answer in a comparison agrees about how much memory the machine has. Results come back in request order.

A request whose modelId names no model on this machine or in the bundled catalog is omitted from results rather than failing the batch. A batch of more than 32 requests is refused with 400 rather than truncated, because a truncated batch reads as a ranking. An empty batch is also a 400. Nothing is reserved, as with POST /v1/models/fit.

post/v1/models/fit/batch

Request

  • Base URL: http://localhost:9998
  • URL: http://localhost:9998/v1/models/fit/batch
  • Auth: HTTP bearer

Request body

modelIdstring required
allocatedTokensinteger nullable

KV capacity in tokens the backend allocates. Not the tokens a request prepares.

streamsinteger nullable

Concurrent sequences sharing the model. KV scales with this.

keyElementBytesinteger nullable

Bytes per stored key element (2 for F16, 1 for an 8-bit KV cache).

valueElementBytesinteger nullable

Bytes per stored value element.

modestring nullable

Where the allocation lives: fullGpu or cpuOffload. Tolerate unknown values.

Response

Advisory verdicts, in request order

Changes

No recorded changes to this operation across all 1 revision of this API.