input_data_pathstring nullable
Path to the input file. The input file should be a JSON file of type List[CreateBatchCompletionsRequestContent].
output_data_pathstring required
Path to the output file. The output file will be a JSON file of type List[CompletionOutput].
labelsobject
Labels to attach to the batch inference job.
data_parallelisminteger nullable
Number of replicas to run the batch inference. More replicas are slower to schedule but faster to inference.
max_runtime_secinteger nullable
Maximum runtime of the batch inference in seconds. Default to one day.
prioritystring nullable
Priority of the batch inference job. Default to None.
gpusinteger nullable
Number of GPUs to use for the batch inference.
gpu_type'nvidia-tesla-t4' | 'nvidia-ampere-a10' | 'nvidia-ampere-a100' | 'nvidia-ampere-a100e' | 'nvidia-hopper-h100' | 'nvidia-hopper-h100-1g20gb' | 'nvidia-hopper-h100-3g40gb'
Lists allowed GPU types for Launch.
nodes_per_workerinteger nullable
Number of nodes per worker for the batch inference.