---
title: "Start a long-running command as a detached background job"
method: POST
path: "/api/v1/jobs"
---

# Start a long-running command as a detached background job

`POST /api/v1/jobs`

## Request

- Base URL: `https://aicommander.dev`
- URL: `https://aicommander.dev/api/v1/jobs`
- Auth: one of:
  - none
  - HTTP bearer

Spawns `command` on the machine identified by `code` as a DETACHED job and returns as soon as it is running — it does NOT wait for the work to finish. USE THIS INSTEAD OF /api/v1/exec for anything expected to take more than a few minutes (ML training, fine-tuning, dataset preparation, large downloads, long builds, batch rendering). Reason: /api/v1/exec is hard-KILLED at 1 hour of wall-clock time, so a training loop dies mid-run and hours of GPU time are lost; and its reply is TRUNCATED at 1 MiB of total output, so a run printing per-step loss loses exactly the log you wanted (the byte cap only asks the machine, best-effort, to stop the command — it usually loses that race and the command keeps running unseen, which is worse, not better). A job has neither cap: its stdout+stderr are written to a file ON THE MACHINE (capped at 256 MiB, after which the agent stops recording but does NOT kill the job), and it keeps running through a network drop and the client going away. A job outlives the AGENT PROCESS on every platform; what differs is what can still take it down, so check `platform` before committing a multi-hour run. On macOS the job reparents to PID 1, which puts it out of reach of ANYTHING aimed at the app — a crash, a hard kill, even an explicit kill of the whole process tree; that makes a Mac the safest place for an unattended run. On Windows the job survives the agent process dying BY ITSELF — a crash, or a kill aimed at that one process (`taskkill /F /IM "AI Commander.exe"`, no `/T`) — measured running straight through such a kill with no gap in its output; what it does NOT survive is a TREE kill: Task Manager's 'End task', `taskkill /T`, or an installer that stops the app and everything it started, because Windows never reparents. TREAT AN AUTO-UPDATE AS A TREE KILL unless you know that machine's installer does otherwise — the silent updater runs the installer, which stops the running app before replacing its files — so make a long unattended Windows run resumable and re-check GET /api/v1/jobs/{id} afterwards. On Linux a job started by an agent that already has this feature is launched into its own transient systemd scope (`aic-job-<jobId>.scope`), outside the agent service's control group, so stopping or restarting the service — including an agent upgrade — leaves it running; measured running gaplessly straight through a `systemctl restart` that killed a control job spawned the old way. That path requires a systemd host with the agent running as root; without both, jobs stay in the service's control group and a restart still ends them. It does not reach backwards either: a job that was already running when the agent was upgraded TO that version is in no scope, and the restart the upgrade itself performs is exactly what ends it, so 'an upgrade is safe' holds only for jobs started after it — list the machine's running jobs and finish or checkpoint them before upgrading a Linux box. Poll it with GET /api/v1/jobs/{id}, read its output with GET /api/v1/jobs/{id}/logs, stop it with DELETE /api/v1/jobs/{id}. A job runs with exactly the rights /api/v1/exec has on that machine; `elevated` is NOT supported and an explicit `elevated:true` is refused rather than silently downgraded. Authorization is identical to /api/v1/exec. Optional `on_complete` asks the relay for one signed, best-effort POST to your own HTTPS endpoint when the job ends (or a deadline passes) so a caller such as a CI pipeline can exit right after the start instead of polling — see the JobOnComplete and JobWebhookDelivery schemas.

## Request body

- object
  - `code` string, required: How the user named the machine, passed exactly as given: an AIC- session code, or — with an API key — a saved alias/hostname (e.g. 'wearfits-m3').
  - `command` string, required: Shell command to run as the job, capped at 32,768 characters. WHICH SHELL DEPENDS ON THE MACHINE'S OS — read `platform` first: POSIX machines ('darwin'/'linux') run it via /bin/sh -c, Windows machines ('win32') via cmd.exe, where `;` is not a separator (`echo a ; echo b` prints the rest as literal text and exits 0 — a silent false success), POSIX tools like `ls` do not exist, heredocs are a syntax error, and a multi-line command is rejected: chain steps with `&&` on one line and write script files via `powershell -NoProfile -Command "..."`. A job's command runs from a BATCH FILE rather than cmd.exe's command line, so `%` follows batch rules here: double a `for` loop's variable (`for /L %%i in (1,1,3) do @echo tick %%i`), because the `%i` that works in /api/v1/exec dies in the job's first second with `i was unexpected at this time.` and an exit code. The job inherits no state from earlier calls, and its stdin is closed — use absolute paths and non-interactive flags.
  - `cwd` string: Working directory, capped at 4,096 characters. MUST be an absolute path that already exists on the machine (a relative or '~'-prefixed value, or a missing directory, fails the start). Defaults to a per-job workspace directory the machine creates.
  - `env` object: Extra environment variables for the job: at most 128 entries, with keys up to 256 characters and string values up to 8,192 characters. Non-string or oversized values reject the request with 400; they are not dropped or coerced. Set HF_HOME / TORCH_HOME here so model weights land somewhere with space instead of the agent account's home directory.
  - `name` string: Short human-readable label (maximum 64 characters), so the job is recognisable later in GET /api/v1/jobs. The machine generates one if omitted.
  - `gpu_index` integer: Reserve this NVIDIA device (the `index` from the machine's GPU list) exclusively for the job: the machine takes an exclusive lock and sets CUDA_VISIBLE_DEVICES accordingly. A second job asking for the same card is refused with 409 `gpu_busy` rather than both runs OOM-ing. Must be an integer, and — when the machine's GPU list is known — must name a card that is actually on it: an out-of-range index is refused with 400 `invalid_request` instead of starting a phantom job whose CUDA_VISIBLE_DEVICES points at nothing and which then fails deep inside the training script. Take the index from the `gpus` array that GET /api/v1/status returns.
  - `on_complete` JobOnComplete: Optional completion webhook: when the job ends — or when `deadline_seconds` passes without an end event — the relay sends ONE small, signed POST to `url` (the payload is the `JobWebhookDelivery` schema). BEST EFFORT, not guaranteed and not exactly-once: at most 3 attempts, at 0, 1 and 5 minutes after the event, then it stops — poll GET /api/v1/jobs/{id} yourself if nothing arrived. Delivery is best effort — at most 3 attempts (0, 1 and 5 minutes after the event), then it stops; duplicates are possible (rarely, when relay storage fails), so de-duplicate by `AIC-Delivery-Id`; poll `GET /api/v1/jobs/{id}` if you must be sure. Success is any 2xx; a network error, timeout (10 s), 5xx, 408 or 429 is retried within those 3 attempts; any other 4xx is permanent (no further attempts); redirects are NOT followed (any 3xx counts as a failure); the response body is never read. There are no custom headers, credentials or body templates: the payload is a closed set of completion metadata and never contains the command, logs, job name or machine alias. The URL is treated as a secret (n8n-style webhook URLs carry one in the path): it is stored encrypted only until delivery finishes or fails, then erased, and is never returned by the API — only its host. REQUIRES an account credential (API key or OAuth) and agent 1.5.0 or newer (job events support) on the target machine; the field is never silently ignored. Refusals specific to `on_complete` (body `{ok:false, error}`, plus `reason` where noted), all BEFORE the job starts: 400 with no `reason` for an invalid `on_complete` (not an object, unknown key, bad or non-canonical URL, bad `deadline_seconds`) — a fixed message that never echoes the URL; 403 `forbidden` for an anonymous caller, when the feature is not enabled on the relay, or when the machine's agent is too old (update it, or start without `on_complete`); 429 `rate_limited` at 100 in-flight webhooks per account or 500 per account and target host per day; 503 `unavailable` when the relay's webhook delivery is not configured; 502 `error` with the message "Could not record the job webhook; the job was not started." when the webhook record could not be stored (nothing was started). A start whose answer was lost is 503 `outcome_unknown` — retry it only when `retrySafe` is true, otherwise check GET /api/v1/jobs first. On a deduplicated retry (`job.deduplicated: true`) this request's `on_complete` is NOT registered and the reply carries `webhook: {state: "not_armed", reason: "deduplicated"}` — only the original start's webhook, if any, fires. Nothing is sent (not even `job.deadline`) when the machine's code is reset, the machine is unlinked or blocked for the account, or the account is deleted; an attempt that is more than about 15 minutes late (e.g. a queue backlog) is dropped as failed rather than sent. Verify every delivery as described in `JobWebhookDelivery`.
    - `url` string, uri, required: Receiver URL. Strict rules: scheme `https` only; port 443 only (an explicit other port — e.g. self-hosted n8n on 5678 — is refused); no `user:pass@` credentials; no `#fragment`; no IP literals (v4 or v6); no `localhost`, `*.local`, `*.localhost`, `*.internal` or dotless hosts; no AI Commander's own domains; no control characters; at most 2048 characters. It must already be in CANONICAL form — exactly equal to WHATWG `new URL(url).href` (lower-case scheme and host, punycode for IDN, no default port, percent-encoded path, `/` path when empty); a non-canonical string is refused rather than rewritten, so the string you send is byte-for-byte the string the signature binds to.
    - `deadline_seconds` integer: Seconds after the job is started (counted from when the relay records the webhook, i.e. just before the start) after which, if no end event has arrived, a `job.deadline` delivery is sent instead. Default 86400 (24 h); 60 s to 604800 (7 days). `job.deadline` says nothing about the job's state — it only means no end event reached the relay in time. If the relay could not record the job id (rare: a storage failure right after the job started) and the job has still not finished at the deadline, no `job.deadline` is sent — poll GET /api/v1/jobs/{id}.
  - `operation_id` string: Idempotency key you choose. A retry with the same `operation_id` returns the job the first attempt started (`job.deduplicated: true`) instead of starting a second one — also after that job has ended, so new work needs a new id. The same id with a different command, cwd, gpu_index or locks is refused 409 `operation_conflict` (with `jobId`); a retry racing the first start's spawn is 409 `operation_in_progress` (retry shortly). The id is remembered only while the job record exists (about 7 days after the job ends); after that it is forgotten and would start the work again. When a start's answer is lost the reply is 503 `outcome_unknown` with `retrySafe: true`: repeat the same request. Scoped to your account (or to the session code) and to this machine. Requires an agent that supports it; an older one is refused 403 rather than ignoring the key.
  - `locks` string[]: Named resources the job holds exclusively while it runs. All are taken or none; a held one refuses 409 `resource_busy` with `lock` and `heldBy`, nothing queues. Per machine, shared by every account on it, released when the job ends or is cancelled. Duplicates are ignored.

## Response `200`

The job's state after the call. When the request carried `on_complete`, the reply also carries `webhook` (the URL itself is never returned).

- object
  - `ok` true, required
  - `job` JobSummary, required: A detached job's state as seen by a caller.
    - `jobId` string, required: 16 hex characters, generated by the machine (never caller-supplied).
    - `name` string, required: The label you passed, or one the machine generated.
    - `status` 'running' | 'exited' | 'unknown', required: `running` = the process was alive when the machine looked. `exited` = the exit code is authoritative. `unknown` = the process is gone and no exit code was recorded — an agent restart, a `SIGKILL`, the OOM killer, an escalated cancellation, or any cancellation on Windows (`taskkill /F`) all produce it, since no exit marker is written; the outcome is genuinely unknowable and must never be reported as success.
    - `exitCode` integer, nullable, required: Non-null only for `exited`.
    - `startedAt` integer, required: Epoch milliseconds.
    - `endedAt` integer, nullable, required: Epoch milliseconds the job ended, or null while it is still running. NOT null for `unknown` jobs: one that recorded no exit still gets a timestamp, but an INFERRED one — see endedAtApproximate, and check that flag before presenting this value as the moment the job finished.
    - `endedAtApproximate` boolean: Present and true when `endedAt` is an ESTIMATE, not an observed instant. A job that recorded no exit marker (SIGKILL, the OOM killer, an escalated cancel, any Windows cancel, the agent going down) has no true end time, so the machine infers one from the last write to its output log — up to ~2 minutes late on Windows, where NTFS defers last-write-time updates while the log is held open. ABSENT means the timestamp was observed and is exact; absent is also what an agent older than 1.0.50 sends for every job. Never quote an approximate value as the finish time, and never measure a duration from one without saying it is approximate.
    - `unknownReason` 'cancelled' | 'killed_by_signal' | 'job_script_removed' | 'vanished_no_output_windows' | 'vanished_no_output_posix' | 'vanished_with_output_windows' | 'vanished_with_output_posix' | 'undetermined': Present ONLY on `unknown` jobs, and only when the machine recorded one: WHAT THE MACHINE OBSERVED about the ending, captured at the moment it settled the job and never re-derived afterwards. It does NOT soften the status — `unknown` still means no exit code was recorded and success may never be assumed. Every token names evidence, not a cause. `cancelled` = someone asked for this job to stop. `killed_by_signal` = the machine watched the job's shell end and it ended on a signal, the only one of these backed by an observed death. `job_script_removed` = the per-job script files were gone from the job's directory (on Windows, the signature of security software quarantining them mid-run — see https://aicommander.dev/antivirus). `vanished_no_output_*` / `vanished_with_output_*` = nothing watched the ending; the scripts were intact and the log was empty, or the log had output in it. A `vanished_*` job may have been killed OR may have completed normally after bypassing the exit marker (a command that `exec`s away or overrides the wrapper's exit trap) — the evidence does not say which. `undetermined` = the machine could not read the job's own directory. ABSENT is normal and means the machine said nothing: an agent older than 1.2.2, or a token this relay does not recognise (which is dropped rather than passed on). Treat an absent value as no information, never as an error, and do not assume this list is closed — a newer agent may add a token, which arrives as absence until the relay learns it.
    - `gpuIndex` integer, nullable, required: The card reserved for this job, or null if none was requested.
    - `logBytes` integer, required: Current size of the job's output log, so you can page it without a probe read.
    - `truncated` boolean, required: True once the log hit its 256 MiB cap and the agent stopped appending. The job itself was NOT killed.
    - `command` string: The job's command line — present ONLY when the request asked for it with include_command=true.
    - `operationId` string: The `operation_id` the job was started with, if any.
    - `locks` string[]: The named locks the job took, if any.
    - `deduplicated` boolean: Present and true on a start answered with the job an earlier start under the same `operation_id` created: nothing new was started.
  - `webhook` union
    - JobWebhookArmed: Present on the POST /api/v1/jobs reply when `on_complete` was accepted. The full URL is never echoed back. There is no webhook state on GET /api/v1/jobs/{id}; keep `deadlineAt` from this reply.
      - `state` 'armed', required
      - `host` string, required: Host part of the receiver URL.
      - `deadlineAt` integer, required: Epoch MILLISECONDS (a number, not an ISO string) after which `job.deadline` is sent if no end event arrived.
    - JobWebhookNotArmed: Present instead of `JobWebhookArmed` when the start was a deduplicated retry (`job.deduplicated: true`): THIS request's `on_complete` was not registered. Only the webhook from the ORIGINAL start under the same `operation_id` (if it had one) will fire; otherwise poll GET /api/v1/jobs/{id}.
      - `state` 'not_armed', required
      - `reason` 'deduplicated', required
      - `message` string, required

## Other responses

- `400`: Missing/invalid fields. On the job endpoints this is also how the MACHINE reports a request it can never run (`reason: "invalid_request"` — an empty command, a relative `cwd`, a `gpu_index` that names no card): the request itself is at fault, so correct the field the message names rather than retrying it.
- `401`: A token was supplied but is not a valid API key/admin/OAuth token.
- `403`: Resolved but refused. Causes include expired anonymous access; Free/anonymous file transfer; `reason: plan_device_limit`; independent operator block/approval; or `error: reactivation_required`. For plan_device_limit, delete an older saved machine to promote the next or upgrade to Pro. For file transfer, use Pro. Do not conflate either with operator approval.
- `404`: Session code or alias not found (anonymous callers get a unified message that does not confirm existence).
- `409`: The machine refused the start. `gpu_busy`: the requested GPU is already reserved (`heldBy` names the job that holds it) — poll that job with GET /api/v1/jobs/{id}, pick another card from the machine's GPU list, or start the job without `gpu_index`. `resource_busy`: a named lock is held (`lock`, `heldBy`) — wait for that job. `operation_conflict`: the `operation_id` already started a DIFFERENT job (`jobId`) — use a new id. `operation_in_progress`: the first start under that `operation_id` is still being spawned — retry the same request shortly. These are expected outcomes, not errors. There is no queue.
- `413`: The encoded request body exceeds this endpoint's byte limit and was rejected before JSON parsing.
- `415`: A JSON request body was sent without an explicit JSON Content-Type. Use `application/json` (parameters such as `charset=utf-8` are allowed); `application/*+json` structured-suffix media types are also accepted.
- `429`: Either the per-IP/global tool-call rate limit was exceeded, or the machine already has the maximum number of jobs running (`reason: "too_many_jobs"`, 32 per machine) — wait for one to finish or cancel one. With `on_complete`, also `reason: "rate_limited"` for the webhook caps: 100 pending webhooks per account, 500 per account and target host per day.
- `502`: `reason: "error"`: the machine answered but failed, so the job was not started (with `on_complete`, also "Could not record the job webhook; the job was not started."). A start whose answer was lost is NOT a 502: it is 503 `reason: "outcome_unknown"` (see 503 and `retrySafe`).
- `503`: `reason: "unavailable"`: the agent is offline or never answered, or (with `on_complete`) webhook delivery is not configured on the relay. `reason: "outcome_unknown"`: the start was sent but its answer was lost, so the job may or may not be running; `retrySafe: true` (the request carried `operation_id`) means repeat the same request, `false` means check GET /api/v1/jobs before any rerun.

---

[API](https://skmtc.dev/aicommander/apis/ai-commander-remote-shell-and-long-running-jobs-api.md) · [All operations](https://skmtc.dev/aicommander/apis/ai-commander-remote-shell-and-long-running-jobs-api/llms.txt) · [OpenAPI document](https://skmtc.dev/aicommander/apis/ai-commander-remote-shell-and-long-running-jobs-api/revisions/a6bb89f779a1?raw)