Models

Advisory memory verdict for one model

Forecasts the memory one model would occupy on this machine and compares it with the host budget read at request time. This is the same answer model_list fit:true reads.

Advisory only: nothing is reserved. Model management and GPU admission recheck at real load time, so a good verdict describes the moment the budget was read and is not a promise about now. Every workload field is optional; an omitted field takes the product default, and the workload actually evaluated is echoed in estimate.workload.

post/v1/models/fit

Request

  • Base URL: http://localhost:9998
  • URL: http://localhost:9998/v1/models/fit
  • Auth: HTTP bearer

Request body

modelIdstring required
allocatedTokensinteger nullable

KV capacity in tokens the backend allocates. Not the tokens a request prepares.

streamsinteger nullable

Concurrent sequences sharing the model. KV scales with this.

keyElementBytesinteger nullable

Bytes per stored key element (2 for F16, 1 for an 8-bit KV cache).

valueElementBytesinteger nullable

Bytes per stored value element.

modestring nullable

Where the allocation lives: fullGpu or cpuOffload. Tolerate unknown values.

Response

Advisory verdict for the model

modelIdstring required
verdictstring required

good (the ceiling fits with the presentation margin), tight (fits below it), oversize (the floor already exceeds the budget) or unknown. Tolerate unknown values.

gpuAdmitsboolean nullable required

Whether GPU admission would admit the estimate's floor without evicting anything; null when no GPU answer was obtainable. Not a reservation.

limitsstring[] required

Evidence the estimator could not use for this answer: artifactNotOnDisk, layoutUnpublished, layoutNotUniform, historyIdentityInsufficient, workspaceUnmeasured, preparedCountUnavailable, hostTelemetryUnavailable, gpuTelemetryUnavailable. Tolerate unknown values.

Changes

No recorded changes to this operation across all 1 revision of this API.