SCscaleapi

Batch Completions

Changed on
post/v2/batch-completions

Request

  • The document declares no server URL.
  • Auth
    • HTTP basic
    • or OAuth 2
      Password
      Token URL: token

Request body

input_data_pathstring nullable

Path to the input file. The input file should be a JSON file of type List[CreateBatchCompletionsRequestContent].

output_data_pathstring required

Path to the output file. The output file will be a JSON file of type List[CompletionOutput].

labelsobject

Labels to attach to the batch inference job.

data_parallelisminteger nullable

Number of replicas to run the batch inference. More replicas are slower to schedule but faster to inference.

max_runtime_secinteger nullable

Maximum runtime of the batch inference in seconds. Default to one day.

prioritystring nullable

Priority of the batch inference job. Default to None.

gpusinteger nullable

Number of GPUs to use for the batch inference.

gpu_type'nvidia-tesla-t4' | 'nvidia-ampere-a10' | 'nvidia-ampere-a100' | 'nvidia-ampere-a100e' | 'nvidia-hopper-h100' | 'nvidia-hopper-h100-1g20gb' | 'nvidia-hopper-h100-3g40gb'

Lists allowed GPU types for Launch.

nodes_per_workerinteger nullable

Number of nodes per worker for the batch inference.

Example request

{
  "content": [
    {
      "max_tokens": 16,
      "n": 1,
      "suffix": "test.",
      "temperature": 1,
      "top_p": 1,
      "user": "user-1234"
    }
  ],
  "model_config": {
    "model": "mixtral-8x7b-instruct"
  }
}

Response

Successful Response

job_idstring required
input_data_pathstring nullable

Path to the input file. The input file should be a JSON file of type List[CreateBatchCompletionsRequestContent].

output_data_pathstring required

Path to the output file. The output file will be a JSON file of type List[CompletionOutput].

prioritystring nullable

Priority of the batch inference job. Default to None.

status'queued' | 'running' | 'completed' | 'failed' | 'cancelled' | 'unknown' required
created_atstring required
expires_atstring required
completed_atstring nullable required
metadataobject nullable required

Example response

{
  "model_config": {
    "model": "mixtral-8x7b-instruct"
  }
}

Changes

  • ea1812999de9Revision changes
    • warning

      removed the optional property // from the response with the status

    • warning

      removed the optional property // from the response with the status