Batches

Create a batch inference job

Changed on

Submit a batch from an uploaded input file (input_file_id) or inline request objects (requests); provide exactly one of the two. Every record is checked when the batch is created. A record that breaks a rule is rejected and the rest are accepted: the answer counts the rejections in rejected_count, lists the first 100 in rejected_records, and rejected_records_file_id names a file holding all of them. A rejected record never runs, is never billed, counts as failed in request_counts, and is also listed in the batch's error file. When no record can run, the answer is 400 no_valid_records with the same fields under error, and nothing is created. A record carrying openrelay.workflow from an organization without workflow records, or priority expedited where it is turned off, refuses the whole submission with 409 feature_disabled, before any record is checked: a workflow record anywhere in the input wins over input_too_large, a read failure after that record, no_valid_records and every record rejection. A batch has 24 hours from submission (expires_at) to finish: records not done by then come back with the batch_expired error and the batch ends expired. Standard batches bill at the discounted batch rate. Poll it with get, then download results via the output and error file ids. An API key whose spend budget is used up gets 402 key_budget_exceeded; a batch already running finishes, and its billed usage counts toward the key.

Requires the batch.write permission. See Permissions for the roles and API key permissions that include it.

post/v1/batches

Request

  • Base URL
    https://inference.openrelay.inc
  • URL
    https://inference.openrelay.inc/v1/batches
  • Auth
    HTTP bearer

Headers

Idempotency-Keystring

Makes a retried create safe. Choose one value per create you mean to make, 1 to 512 bytes (bytes, not characters), such as a UUID, and send it again on every retry of that create. The first request with a key runs, and a 2xx answer is kept for 24 hours: a repeat with the same key and the same body gets that answer back, with the same status and the header Idempotent-Replayed: true, and creates nothing. A batch create compares the body byte for byte; a VM or pod create compares the fields it decodes, so whitespace and key order do not count. A repeat while the first request is still running answers 409; repeat it a second later. The same key with a different body answers 422. An answer other than a 2xx is not kept, so a retry after a 4xx or a 5xx runs again. A 2xx too large to keep holds the key without it: every repeat answers 409 and creates nothing. Keys are scoped to the organization and the operation.

Request body

completion_window'24h' required

Time budget for the batch; only "24h" is supported.

endpoint'/v1/chat/completions' | '/v1/completions' | '/v1/embeddings' required

The API family every record in the batch calls.

input_file_idstring

Id of an uploaded JSONL input file (from the file upload).

metadataobject

Key-value pairs echoed back on the batch object. Updating metadata later allows at most 32 pairs.

modelstring

The model a record without body.model runs, when it is one batch serves; otherwise it is ignored. The batch's model is the one its first accepted record runs.

priority'standard' | 'expedited'

Scheduling tier. standard (the default) runs at the discounted batch rate. expedited is granted a freeing pool slot ahead of any waiting standard batch, grows the fleet to a tighter drain target, and is billed at the online rate with no batch discount; it is available to every organization unless an operator has disabled it for yours, in which case the request is refused with 409 feature_disabled. Neither tier changes the 24-hour expiry.

Response

The batch was created. rejected_count is how many records the create rejected; when it rejected any, the answer also carries rejected_records and rejected_records_file_id.

cancelled_atinteger
completed_atinteger
completion_window'24h' required
created_atinteger required

Unix timestamp (seconds).

endpointstring required

The API family every record in the batch calls.

error_file_idstring

Present once results are written; fetch its content for failed records, the records rejected at create included.

expired_atinteger
expires_atinteger

When the completion window closes (created_at + 24h).

failed_atinteger
idstring required

Batch id (batch_…).

in_progress_atinteger nullable required

Unix timestamp (seconds) when the batch's first requests started running. Always present and null until then, so an in_progress batch with a null in_progress_at is still queued for GPU capacity.

input_file_idstring required
metadataobject

Your key-value pairs, echoed back unchanged.

modelstring

The model every record runs: the first accepted record's, set when the batch is created.

object'batch' required
output_file_idstring

Present once results are written; fetch its content for successful records.

priority'standard' | 'expedited' required

The scheduling tier this batch runs and is billed at. Batches created before the tier existed read as standard.

rejected_countinteger

Only on the create answer: how many records the create rejected. Read the rejected count here, not from request_counts.failed, which also counts the records that failed running when the create answered after its batch already ran.

rejected_records_file_idstring

Only on the create answer, and only when the create rejected records: the file holding every rejected record (purpose batch_rejections), one BatchRecordRejection per line, in input order. Fetch it with the file content call.

status'validating' | 'in_progress' | 'finalizing' | 'completed' | 'failed' | 'expired' | 'cancelling' | 'cancelled' required

Lifecycle state. validating → in_progress → finalizing → completed | failed | expired; cancelling → cancelled. A batch is in_progress while it waits for GPU capacity too; in_progress_at says when its requests started running.

Changes

  • 7fe59d94a9f5Revision changes
    • info

      added the non-success response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

  • dd53333296b0Revision changes
    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property to the response with the status

    • info

      added the optional property to the response with the status

    • info

      added the optional property to the response with the status

    • info

      added the optional property to the response with the status