--- title: "Submit a batch job" method: POST path: "/v1/batch" tags: ["Batch"] --- # Submit a batch job `POST /v1/batch` ## Request - Base URL: `https://api.spicrawl.com` - URL: `https://api.spicrawl.com/v1/batch` - Auth: HTTP bearer Queues many URLs as one asynchronous job and returns `202` with the job object immediately. Supply either `urls` (same settings for every URL) or `items` (one object per URL, with per-item overrides), never both. Every /v1/scrape parameter at the top level (except `url`) is a job-wide default; an item's own value wins over it, including an explicit `false`. Order of work: validate and route every item (a bad item N fails the whole submission with `Item N:` in `detail`), reserve the dearest possible cost of the whole job against the monthly credit allowance, write the manifest, enqueue. Nothing is charged at submission (`X-Credits-Charged: 0`); items are charged individually, only on success, by the worker. Next step: poll `GET /v1/batch/{batchID}` (the `status_url`) until `status` is `completed`, `failed` or `cancelled`, then page `GET /v1/batch/{batchID}/results`. Results are readable while the job runs, too. Not idempotent: there is no Idempotency-Key. Resubmitting the same body creates a second job that is scraped and billed again. If a `202` carries a warning saying items were not all dispatched, do NOT resubmit — the worker's recovery sweep dispatches them. Limits: request body 1 MiB (1,048,576 bytes); at most 10,000 items per call; job-level parameters at most 65,536 bytes serialised; each item's overrides plus `external_id` at most 8,192 bytes serialised. ## Request body - BatchCreateRequest: The /v1/scrape request object as accepted by batch, minus `url`. Field names, types and validation are exactly ScrapeRequest's (enforced by a parity test); each property below references it. Every item is merged as job-level value, overridden by the item's value (an explicit `false` on an item wins), with the project's dashboard defaults underneath both. Validation happens for every item at submission. What the batch worker executes: each item is one fetch or one render, and the document is returned as `response_format` (`html` by default, `json`, `markdown` or `text`). A field is either honoured on a batch item or refused at submission, never accepted and skipped. Refused with 400 INVALID_PARAMETER, naming every offending field, with nothing queued or charged: `actions`, `screenshot` and its options, `network_capture`, `extract`, `extract_preset`, `ai_extract`, `autoparse`, `links`, `session_id`, `cache`/`cache_ttl`, `original_status`, `allowed_status_codes`, `parse_pdf=true`, `mode=auto`, `response_format=pdf`, and a non-GET `method` with `js_render` or `stealth`. A value that asks for nothing (`screenshot: false`, an empty list) is accepted. A refusal caused by a project default says so in `detail`. Use /v1/scrape for those fields. A batch item is priced when it is accepted, so a bot challenge never escalates it to stronger rendering or retries it on a different exit the way it can a /v1/scrape request with your own `proxy`. - `method` Method (unresolved $ref) - `js_render` JsRender (unresolved $ref) - `stealth` Stealth (unresolved $ref) - `impersonate` Impersonate (unresolved $ref) - `mode` Mode (unresolved $ref) - `engine` Engine (unresolved $ref) - `premium_proxy` PremiumProxy (unresolved $ref) - `proxy_country` ProxyCountry (unresolved $ref) - `proxy` Proxy (unresolved $ref) - `proxy_verify` ProxyVerify (unresolved $ref) - `session_id` SessionId (unresolved $ref) - `sticky_key` StickyKey (unresolved $ref) - `wait` Wait (unresolved $ref) - `wait_for` WaitFor (unresolved $ref) - `wait_for_timeout` WaitForTimeout (unresolved $ref) - `actions` Actions (unresolved $ref) - `block_resources` BlockResources (unresolved $ref) - `headless` Headless (unresolved $ref) - `screenshot` Screenshot (unresolved $ref) - `screenshot_fullpage` ScreenshotFullpage (unresolved $ref) - `screenshot_selector` ScreenshotSelector (unresolved $ref) - `screenshot_format` ScreenshotFormat (unresolved $ref) - `screenshot_quality` ScreenshotQuality (unresolved $ref) - `custom_headers` CustomHeaders (unresolved $ref) - `autoparse` Autoparse (unresolved $ref) - `links` Links (unresolved $ref) - `network_capture` NetworkCapture (unresolved $ref) - `extract` Extract (unresolved $ref) - `extract_preset` ExtractPreset (unresolved $ref) - `ai_extract` AiExtract (unresolved $ref) - `main_content_only` MainContentOnly (unresolved $ref) - `include_tags` IncludeTags (unresolved $ref) - `exclude_tags` ExcludeTags (unresolved $ref) - `parse_pdf` ParsePdf (unresolved $ref) - `response_format` ResponseFormat (unresolved $ref) - `cache` Cache (unresolved $ref) - `cache_ttl` CacheTtl (unresolved $ref) - `max_cost` MaxCost (unresolved $ref) - `original_status` OriginalStatus (unresolved $ref) - `allowed_status_codes` AllowedStatusCodes (unresolved $ref) ## Response `202` Accepted. The manifest is durably stored and the job is `queued`. Read `warnings`: it reports a clamped `concurrency` and any dispatch shortfall. Poll `status_url` next. - BatchJob: The batch job, identical in shape across create, list, get, cancel and close. Poll `status_url` until `status` is terminal (`completed`, `failed`, `cancelled`), then read `results_url` before `results_expire_at`. There is no `credits_held` field: the hold is `estimated_credits`, the spend is `progress.credits_charged`. - `id` string, required: Job id (26-character ULID). Use it in every /v1/batch/{batchID} path. - `project_id` string, required: Project the submitting API key belongs to. - `name` string: The `name` you submitted. Absent when none. - `status` 'queued' | 'running' | 'paused' | 'cancelling' | 'completed' | 'failed' | 'cancelled', required: Job lifecycle. `queued` → `running` → `completed` | `failed`; cancel moves a live job to `cancelling`, then the worker moves it to `cancelled` once in-flight items drain. `paused` is an operator state (no public endpoint pauses). Terminal: `completed`, `failed`, `cancelled` — stop polling. A terminal job returns to `queued` after batchRetry. - `open` boolean: Present and true while the job accepts appended items; it will not complete until you close it. Absent means closed. - `status_url` string, required: Relative path to poll: /v1/batch/{id}. - `results_url` string, required: Relative path of the JSONL results: /v1/batch/{id}/results. - `params` object, required: The job-level scrape parameters as stored (your top-level BatchScrapeParams, without `url`), plus two keys the API sets and you cannot — `credit_budget_micro`, the run ceiling the worker enforces, and `default_proxy_tier` (`direct` or `datacenter`), the exit an item that sets no `proxy`, `premium_proxy`, `proxy_country` or `sticky_key` takes, decided at submission the way `/v1/scrape` decides it. Always present, possibly only those two. - `total_items` integer, required: Number of items in the job. Fixed at submission unless items are appended to an open job. - `concurrency` integer, required: Effective per-job concurrency after clamping. - `priority` integer, required: Effective priority. - `max_attempts` integer, required: Effective attempts per item. - `failure_threshold` integer: Failed-item count that aborts the job. Absent when not set. - `estimated_credits` integer, required: Credits RESERVED against the monthly allowance at submission — the dearest outcome of every item (whole credits, rounded down). It is a hold, not a charge; credits held for items that fail, are cancelled or skipped are returned to the monthly allowance within about a minute of the item finishing, by a background settler. Succeeded items keep their charge. Not increased by appends. Appended items are reserved when they are added, and the append's own total is its `X-Request-Cost`. Retrying failed items (`POST /v1/batch/{id}/retry`) holds their price again first and is refused with 402 if the allowance cannot cover it. - `estimated_credits_micro` integer, required: Same as `estimated_credits` in micro-credits. - `progress` BatchJobProgress, required: Live counters for the job, read from sharded counters. `completed` = succeeded + failed + cancelled + skipped; `remaining` = total − completed. - `total` integer, required: Items in the job (same as `total_items`). Grows when items are appended to an open job. - `completed` integer, required: Items in any terminal status. - `succeeded` integer, required: Items that succeeded. Only these are charged. - `failed` integer, required: Items that failed after all attempts. Re-run them with POST /v1/batch/{batchID}/retry. - `cancelled` integer, required: Items not run because the job was cancelled. - `skipped` integer, required: Items the worker skipped (for example after `failure_threshold` or `credit_budget` stopped the run). - `remaining` integer, required: Items not yet finished. - `percent_complete` number, required: completed / total × 100. - `credits_charged` integer, required: Credits actually charged so far, whole credits rounded down. Sum over succeeded items only. This is the real spend; compare with the job's `estimated_credits` (the hold). - `credits_charged_micro` integer, required: Same as `credits_charged` in micro-credits (1 credit = 1,000,000). Use this for exact accounting. - `bytes` integer, required: Total response bytes of finished items. - `submitted_at` string, date-time, required: When the job was accepted. - `started_at` string, date-time: When the first item started. Absent until then. - `finished_at` string, date-time: When the job reached a terminal status. Cleared again by batchRetry. - `cancel_requested_at` string, date-time: When a cancel was accepted. The job may still be `cancelling` after this while in-flight items drain. - `results_expire_at` string, date-time: Results deadline: submitted_at + 72 h by default. After it, GET .../results answers 410. Download before this time. - `error_code` string: Machine code of a job-level failure (for example the job was aborted). Absent otherwise. - `error_message` string: Human-readable job-level failure reason. - `warnings` string[]: Decisions made on your behalf at submission — a clamped `concurrency`, or items not yet handed to the queue (do NOT resubmit in that case). Present only on the create response. ## Other responses - `400`: The request is invalid. Not retryable; fix the request using `code` and `diagnostics.hint`. - `401`: Missing, invalid, revoked or expired key. Not retryable with the same key. - `402`: `ERR::LIMIT::QUOTA_EXCEEDED`: the organization is out of credits. Not retryable until credits are added. - `403`: The key lacks the scope this route requires (`ERR::AUTH::INSUFFICIENT_SCOPE`), or the action is not permitted. - `413`: The request body exceeds the size limit. - `500`: Internal error. Retryable when `retryable` is true. - `503`: No proxy exit or engine capacity was available. Retryable after `Retry-After`; 0 credits. --- [API](https://skmtc.dev/spicrawl/apis/spicrawl-api.md) · [All operations](https://skmtc.dev/spicrawl/apis/spicrawl-api/llms.txt) · [OpenAPI document](https://skmtc.dev/spicrawl/apis/spicrawl-api/revisions/c4b4c6f97219?raw)