Submit an async job

The longest time in minutes that the image build can take (default 30, maximum 120). The build stops at this limit. Increase it for a build that needs more time, for example a large dependency tree from a slow upstream, or long ahead-of-time compilation. One limit covers the whole build. It includes the context download, your Dockerfile steps, and the push of the finished image to the registry. The push of a large image can take minutes. If the steps finish near the limit, the push fails at the same point on each retry. If you see "publishing the built image timed out", increase this value or make the image smaller. A resubmit does not fix it. A prebuilt image uses the same limit for its import. The import pulls the image and pushes it into the platform registry, so give a large image several minutes. If the limit stops an import while its pull moves at a normal speed, the import fails with no retry, and its failure_reason tells you what to change. We bill only metered build time. We never bill queued or idle time, and a publish timeout is free.

Chip: "h100" (GPU), or a CPU-only size, "cpu-2" (2 reserved cores, 8 GiB RAM) or "cpu-8" (8 reserved NUMA-local cores, 32 GiB RAM) - no GPU, at that size's CPU job rate. CPU jobs fit CPU-bound batch stages (dedup, filtering, tokenization, decompress/convert, downloads) that would idle a GPU; cpu-2 is the honest size for downloads and ingest, cpu-8 for parallel stages. Parallelism comes from submitting one job per shard (submit_jobs fans out a sweep), not from chip_count. Like all jobs it may queue and be preempted; like the CPU nodes it may run far from your GPU nodes, so move data with output_to, contexts, and object storage, not by assuming a LAN. (Named for the accelerator, not the vendor.)

Chips per attempt (1-8). Defaults to 1. GPU chips only - a CPU size is one reserved-core shape, so fan out jobs instead of counting chips. For more than one machine, set chip_count to 8 and set nodes.

Benchmark mode: pin this attempt's GPUs to the model's published sustained clock for reproducible timings, instead of letting them drift with boost/thermal state. Set once at submit and immutable for the job's life (like the image). Physics, stated honestly: the lock is a CEILING, not a guarantee (the hardware still throttles below it under power/thermal pressure), and on an 8-GPU SXM baseboard neighbors' heat couples, so a full 8-GPU shape is the only fully isolated one. Free: no rate change (you pay in your own forgone boost headroom). Default false. Use it for kernel/inference sweeps where noisy timings would poison the comparison.

The command to run in the container (overrides the image's default).

An uploaded large context from create_context. Mutually exclusive with image; composes with inline context (inline overlays it, inline wins).

Dockerfile path within the context (default "Dockerfile").

Override a preflight BLOCK for this submission only (docs/designs/job-checks.md part 2). Today one check blocks: the job's worst-case cost against your remaining spend cap. Use it for a job you know will stop early - the block exists because the alternative is the cap stopping the same job mid-run, after the money is spent. Every override is recorded on the job and read as a report about the check.

Mount your org's shared HuggingFace cache at HF_HUB_CACHE for every attempt: weights any of your org's jobs already pulled on the same machine load from local disk instead of re-downloading into billed time. Opt-in because the cache outlives the job - files written under HF_HUB_CACHE stay for your org's later jobs (never another org's); a default job still leaves nothing behind. Best-effort by design: the cache is bounded per machine, may be cleared under disk pressure, and when it fills the longest-unused models are evicted so each job starts with at least 32 GiB of room - a job that gets no cache just downloads as usual, though a single download larger than that free room can still fill the cache mid-pull (submit unusually large models without the flag). Non-hub files parked there count against the same room, and a cache they fill past what evicting models can free is cleared whole and rebuilds cold. Refused together with an env named HF_HUB_CACHE (the platform sets that variable). Free: no rate change. Default false.

Idempotency key: resubmitting with the same key returns the SAME job (never a duplicate). Reusing a key for a different request is refused.

Use a prebuilt image by ref instead of building (e.g. "nvcr.io/nvidia/pytorch:24.01-py3"). Mutually exclusive with context/context_id. A PRIVATE image works too, with no credential in this call: if an org admin stored a registry connection whose scope covers this ref (list_connections), the platform authenticates the pull itself. Otherwise the pull is anonymous, and a private ref fails with the registry's own "pull access denied" - the fix is a connection, never a token in the ref.

A human-readable label shown in listings.

Max run minutes before the job is stopped (default 60; the ceiling is a per-org limit, 720 by default - see list_limits).

Provider-fault restarts allowed before failing (default 1, ceiling 3). Preemptions do NOT consume this.

The mission this job belongs to (open_mission first - it is create-or-attach, so re-opening a name resumes it): the investigation-level page that rolls up cost, results, and artifacts, and the link you hand the human. Name it for the question you are answering. Required: a call that omits it is refused with the fix.

The model this job downloads, e.g. "Qwen/Qwen3-VL-72B" or "Qwen/Qwen3-VL-72B@main" (docs/designs/job-checks.md part 2). Optional: preflight otherwise reads the --model flag and the from_pretrained literals out of your command. Declaring it is what makes the disk check exact instead of heuristic - and when the model cannot be identified, the check stays silent rather than guessing.

Skip preflight entirely (the fast resubmit loop). Free either way. The job record then says it skipped them, so a later postmortem knows which checks did not run.

Run the job on more than one whole machine. Set nodes from 2 to 8. Your account limit can be lower (see list_limits). Set chip to h100. Set chip_count to 8, or omit it. Set resume to checkpoint. Autoresearch puts the machines on a private InfiniBand network for this job only. Each machine runs one copy of command with 8 GPUs. Each copy gets NODE_RANK (0 to nodes-1), NNODES, MASTER_ADDR and MASTER_PORT. So torchrun --nnodes $NNODES --node-rank $NODE_RANK --nproc-per-node 8 --master-addr $MASTER_ADDR --master-port $MASTER_PORT train.py works as written. Your image must contain libibverbs and the mlx5 provider. The NGC PyTorch images contain them. The job waits while Autoresearch prepares the machines. This usually takes 10 to 45 minutes. You do not pay while the job waits. An attempt bills every GPU on every machine from the moment every machine starts command until the attempt ends. This is also true when a machine fails. Then the job runs again from the checkpoint, at most max_restarts more times, and each run bills. A GPU-minute costs the h100 list rate, as on an h100-8 node, with no job discount. A promotional discount on nodes does not apply. Fridays and weekends lower it as they do for a node. The copy with NODE_RANK 0 is rank 0. Save checkpoints from rank 0 only. Rank 0 writes the whole checkpoint to $GMN_CHECKPOINT_DIR. Then rank 0 calls POST $GMN_METADATA_URL/v1/checkpoint. On a new attempt, every machine finds that checkpoint in $GMN_CHECKPOINT_DIR. Only rank 0 keeps $GMN_RESULT_PATH and $GMN_OUTPUT_DIR. Set max_duration_minutes to 1200 or less. A job with nodes cannot use tasks, profiling, yield or on_fail reschedule.

Benchmark mode's other half: unlock GPU profiler counters (CUPTI - what ncu needs) for the attempt. Whole-machine shapes only (chip_count: 8): GPU performance counters are a cross-tenant side channel, so on a shared machine they are never for sale. The attempt places only on a profiling-capable machine (a small cohort - expect extra queue wait when it's busy). Free: no rate change (you pay in shape). Immutable at submit, like clock_lock - kernel-search jobs typically set both. Default false. Unlike create_node, a job runs YOUR image, so bring your own profiler: it must ship nsight-compute 2025 or newer, or ncu fails on current drivers with an opaque "Failed to prepare kernel for profiling" that no unlocked machine can fix. Basing on CUDA 12.9+ (or apt-installing cuda-nsight-compute) clears it.

How long the job may wait in the queue before expiring (default 4320 = 3 days).

Resumable placement: pass "checkpoint" to declare the command resume-aware - if $GMN_CHECKPOINT_DIR is non-empty at start, it resumes from it, and it checkpoints on its own cadence (write the dir, then POST /v1/checkpoint on the metadata service). The scheduler may then start the job in shorter capacity windows: an allocation boundary becomes a planned stop-and-resume (never consuming max_restarts), and a severed attempt bills only to its last checkpoint. Any job with a committed capture may also POST /v1/yield to end its attempt and resume from the slot on demand (bills the full span - a restore, never a discount). Omit and the job runs as one attempt - unless a prior job in your org ran this exact image with this exact command and checkpointed, in which case it inherits "checkpoint" (get_job shows resume_source: "inferred" and the job it learned from). Pass "none" to refuse that inheritance. A job with nodes must pass checkpoint. See nodes for how that job bills.

How long one leased task may run before the pool takes it back (default 15, max 120). Size it above your slowest unit, or renew with POST /v1/tasks/{id}/extend.

How many times one task may be leased before it is recorded failed (default 3). A failed task does not fail the job.

How many machines may work the set at once (default 4, max 32). We start them as the backlog warrants and stop them as it drains, so this is a ceiling on spend, not a reservation: a set of 20 units never starts 32 machines.

Keep the task set OPEN so you can add_tasks as you go. The job keeps running while the set is open and finishes once you seal_tasks and the pool drains. Omit for a set that is complete at submit.

The resource type, when the result is one resource.