Start a long-running command as a detached background job

Spawns command on the machine identified by code as a DETACHED job and returns as soon as it is running — it does NOT wait for the work to finish. USE THIS INSTEAD OF /api/v1/exec for anything expected to take more than a few minutes (ML training, fine-tuning, dataset preparation, large downloads, long builds, batch rendering). Reason: /api/v1/exec is hard-KILLED at 1 hour of wall-clock time, so a training loop dies mid-run and hours of GPU time are lost; and its reply is TRUNCATED at 1 MiB of total output, so a run printing per-step loss loses exactly the log you wanted (the byte cap only asks the machine, best-effort, to stop the command — it usually loses that race and the command keeps running unseen, which is worse, not better). A job has neither cap: its stdout+stderr are written to a file ON THE MACHINE (capped at 256 MiB, after which the agent stops recording but does NOT kill the job), and it keeps running through a network drop and the client going away. A job outlives the AGENT PROCESS on every platform; what differs is what can still take it down, so check platform before committing a multi-hour run. On macOS the job reparents to PID 1, which puts it out of reach of ANYTHING aimed at the app — a crash, a hard kill, even an explicit kill of the whole process tree; that makes a Mac the safest place for an unattended run. On Windows the job survives the agent process dying BY ITSELF — a crash, or a kill aimed at that one process (taskkill /F /IM "AI Commander.exe", no /T) — measured running straight through such a kill with no gap in its output; what it does NOT survive is a TREE kill: Task Manager's 'End task', taskkill /T, or an installer that stops the app and everything it started, because Windows never reparents. TREAT AN AUTO-UPDATE AS A TREE KILL unless you know that machine's installer does otherwise — the silent updater runs the installer, which stops the running app before replacing its files — so make a long unattended Windows run resumable and re-check GET /api/v1/jobs/{id} afterwards. On Linux a job started by an agent that already has this feature is launched into its own transient systemd scope (aic-job-<jobId>.scope), outside the agent service's control group, so stopping or restarting the service — including an agent upgrade — leaves it running; measured running gaplessly straight through a systemctl restart that killed a control job spawned the old way. That path requires a systemd host with the agent running as root; without both, jobs stay in the service's control group and a restart still ends them. It does not reach backwards either: a job that was already running when the agent was upgraded TO that version is in no scope, and the restart the upgrade itself performs is exactly what ends it, so 'an upgrade is safe' holds only for jobs started after it — list the machine's running jobs and finish or checkpoint them before upgrading a Linux box. Poll it with GET /api/v1/jobs/{id}, read its output with GET /api/v1/jobs/{id}/logs, stop it with DELETE /api/v1/jobs/{id}. A job runs with exactly the rights /api/v1/exec has on that machine; elevated is NOT supported and an explicit elevated:true is refused rather than silently downgraded. Authorization is identical to /api/v1/exec. Optional on_complete asks the relay for one signed, best-effort POST to your own HTTPS endpoint when the job ends (or a deadline passes) so a caller such as a CI pipeline can exit right after the start instead of polling — see the JobOnComplete and JobWebhookDelivery schemas.

post/api/v1/jobs

Request

  • Base URL: https://aicommander.dev
  • URL: https://aicommander.dev/api/v1/jobs
  • Auth: one of:
    • none
    • HTTP bearer

Request body

codestring required

How the user named the machine, passed exactly as given: an AIC- session code, or — with an API key — a saved alias/hostname (e.g. 'wearfits-m3').

commandstring required

Shell command to run as the job, capped at 32,768 characters. WHICH SHELL DEPENDS ON THE MACHINE'S OS — read platform first: POSIX machines ('darwin'/'linux') run it via /bin/sh -c, Windows machines ('win32') via cmd.exe, where ; is not a separator (echo a ; echo b prints the rest as literal text and exits 0 — a silent false success), POSIX tools like ls do not exist, heredocs are a syntax error, and a multi-line command is rejected: chain steps with && on one line and write script files via powershell -NoProfile -Command "...". A job's command runs from a BATCH FILE rather than cmd.exe's command line, so % follows batch rules here: double a for loop's variable (for /L %%i in (1,1,3) do @echo tick %%i), because the %i that works in /api/v1/exec dies in the job's first second with i was unexpected at this time. and an exit code. The job inherits no state from earlier calls, and its stdin is closed — use absolute paths and non-interactive flags.

cwdstring

Working directory, capped at 4,096 characters. MUST be an absolute path that already exists on the machine (a relative or '~'-prefixed value, or a missing directory, fails the start). Defaults to a per-job workspace directory the machine creates.

envobject

Extra environment variables for the job: at most 128 entries, with keys up to 256 characters and string values up to 8,192 characters. Non-string or oversized values reject the request with 400; they are not dropped or coerced. Set HF_HOME / TORCH_HOME here so model weights land somewhere with space instead of the agent account's home directory.

namestring

Short human-readable label (maximum 64 characters), so the job is recognisable later in GET /api/v1/jobs. The machine generates one if omitted.

gpu_indexinteger

Reserve this NVIDIA device (the index from the machine's GPU list) exclusively for the job: the machine takes an exclusive lock and sets CUDA_VISIBLE_DEVICES accordingly. A second job asking for the same card is refused with 409 gpu_busy rather than both runs OOM-ing. Must be an integer, and — when the machine's GPU list is known — must name a card that is actually on it: an out-of-range index is refused with 400 invalid_request instead of starting a phantom job whose CUDA_VISIBLE_DEVICES points at nothing and which then fails deep inside the training script. Take the index from the gpus array that GET /api/v1/status returns.

operation_idstring

Idempotency key you choose. A retry with the same operation_id returns the job the first attempt started (job.deduplicated: true) instead of starting a second one — also after that job has ended, so new work needs a new id. The same id with a different command, cwd, gpu_index or locks is refused 409 operation_conflict (with jobId); a retry racing the first start's spawn is 409 operation_in_progress (retry shortly). The id is remembered only while the job record exists (about 7 days after the job ends); after that it is forgotten and would start the work again. When a start's answer is lost the reply is 503 outcome_unknown with retrySafe: true: repeat the same request. Scoped to your account (or to the session code) and to this machine. Requires an agent that supports it; an older one is refused 403 rather than ignoring the key.

locksstring[]

Named resources the job holds exclusively while it runs. All are taken or none; a held one refuses 409 resource_busy with lock and heldBy, nothing queues. Per machine, shared by every account on it, released when the job ends or is cancelled. Duplicates are ignored.

Example request

{
  "code": "AIC-7K3P-WX9M-RTBN",
  "command": "./.venv/bin/python train.py",
  "cwd": "/home/u/aic-jobs/mytrain",
  "env": {
    "HF_HOME": "/home/u/aic-jobs/mytrain/.cache/hf"
  },
  "name": "mytrain",
  "on_complete": {
    "url": "https://hooks.example.com/aic/3f9c2e7b",
    "deadline_seconds": 86400
  },
  "operation_id": "train-resnet-run1",
  "locks": [
    "dataset:imagenet"
  ]
}

Response

The job's state after the call. When the request carried on_complete, the reply also carries webhook (the URL itself is never returned).

oktrue required

Example response

{
  "job": {
    "jobId": "9f2c1b7ad4e05386"
  },
  "webhook": {
    "host": "hooks.example.com"
  }
}

Changes

No recorded changes to this operation across all 1 revision of this API.