Batch

Submit a batch job

Queues many URLs as one asynchronous job and returns 202 with the job object immediately. Supply either urls (same settings for every URL) or items (one object per URL, with per-item overrides), never both. Every /v1/scrape parameter at the top level (except url) is a job-wide default; an item's own value wins over it, including an explicit false.

Order of work: validate and route every item (a bad item N fails the whole submission with Item N: in detail), reserve the dearest possible cost of the whole job against the monthly credit allowance, write the manifest, enqueue. Nothing is charged at submission (X-Credits-Charged: 0); items are charged individually, only on success, by the worker.

Next step: poll GET /v1/batch/{batchID} (the status_url) until status is completed, failed or cancelled, then page GET /v1/batch/{batchID}/results. Results are readable while the job runs, too.

Not idempotent: there is no Idempotency-Key. Resubmitting the same body creates a second job that is scraped and billed again. If a 202 carries a warning saying items were not all dispatched, do NOT resubmit — the worker's recovery sweep dispatches them.

Limits: request body 1 MiB (1,048,576 bytes); at most 10,000 items per call; job-level parameters at most 65,536 bytes serialised; each item's overrides plus external_id at most 8,192 bytes serialised.

post/v1/batch

Request

  • Base URL: https://api.spicrawl.com
  • URL: https://api.spicrawl.com/v1/batch
  • Auth: HTTP bearer

Request body

methodMethod (unresolved $ref)
js_renderJsRender (unresolved $ref)
stealthStealth (unresolved $ref)
impersonateImpersonate (unresolved $ref)
modeMode (unresolved $ref)
engineEngine (unresolved $ref)
premium_proxyPremiumProxy (unresolved $ref)
proxy_countryProxyCountry (unresolved $ref)
proxyProxy (unresolved $ref)
proxy_verifyProxyVerify (unresolved $ref)
session_idSessionId (unresolved $ref)
sticky_keyStickyKey (unresolved $ref)
waitWait (unresolved $ref)
wait_forWaitFor (unresolved $ref)
wait_for_timeoutWaitForTimeout (unresolved $ref)
actionsActions (unresolved $ref)
block_resourcesBlockResources (unresolved $ref)
headlessHeadless (unresolved $ref)
screenshotScreenshot (unresolved $ref)
screenshot_fullpageScreenshotFullpage (unresolved $ref)
screenshot_selectorScreenshotSelector (unresolved $ref)
screenshot_formatScreenshotFormat (unresolved $ref)
screenshot_qualityScreenshotQuality (unresolved $ref)
custom_headersCustomHeaders (unresolved $ref)
autoparseAutoparse (unresolved $ref)
linksLinks (unresolved $ref)
network_captureNetworkCapture (unresolved $ref)
extractExtract (unresolved $ref)
extract_presetExtractPreset (unresolved $ref)
ai_extractAiExtract (unresolved $ref)
main_content_onlyMainContentOnly (unresolved $ref)
include_tagsIncludeTags (unresolved $ref)
exclude_tagsExcludeTags (unresolved $ref)
parse_pdfParsePdf (unresolved $ref)
response_formatResponseFormat (unresolved $ref)
cacheCache (unresolved $ref)
cache_ttlCacheTtl (unresolved $ref)
max_costMaxCost (unresolved $ref)
original_statusOriginalStatus (unresolved $ref)
allowed_status_codesAllowedStatusCodes (unresolved $ref)

Response

Accepted. The manifest is durably stored and the job is queued. Read warnings: it reports a clamped concurrency and any dispatch shortfall. Poll status_url next.

idstring required

Job id (26-character ULID). Use it in every /v1/batch/{batchID} path.

project_idstring required

Project the submitting API key belongs to.

namestring

The name you submitted. Absent when none.

status'queued' | 'running' | 'paused' | 'cancelling' | 'completed' | 'failed' | 'cancelled' required

Job lifecycle. queued → running → completed | failed; cancel moves a live job to cancelling, then the worker moves it to cancelled once in-flight items drain. paused is an operator state (no public endpoint pauses). Terminal: completed, failed, cancelled — stop polling. A terminal job returns to queued after batchRetry.

openboolean

Present and true while the job accepts appended items; it will not complete until you close it. Absent means closed.

status_urlstring required

Relative path to poll: /v1/batch/{id}.

results_urlstring required

Relative path of the JSONL results: /v1/batch/{id}/results.

paramsobject required

The job-level scrape parameters as stored (your top-level BatchScrapeParams, without url), plus two keys the API sets and you cannot — credit_budget_micro, the run ceiling the worker enforces, and default_proxy_tier (direct or datacenter), the exit an item that sets no proxy, premium_proxy, proxy_country or sticky_key takes, decided at submission the way /v1/scrape decides it. Always present, possibly only those two.

total_itemsinteger required

Number of items in the job. Fixed at submission unless items are appended to an open job.

concurrencyinteger required

Effective per-job concurrency after clamping.

priorityinteger required

Effective priority.

max_attemptsinteger required

Effective attempts per item.

failure_thresholdinteger

Failed-item count that aborts the job. Absent when not set.

estimated_creditsinteger required

Credits RESERVED against the monthly allowance at submission — the dearest outcome of every item (whole credits, rounded down). It is a hold, not a charge; credits held for items that fail, are cancelled or skipped are returned to the monthly allowance within about a minute of the item finishing, by a background settler. Succeeded items keep their charge. Not increased by appends. Appended items are reserved when they are added, and the append's own total is its X-Request-Cost. Retrying failed items (POST /v1/batch/{id}/retry) holds their price again first and is refused with 402 if the allowance cannot cover it.

estimated_credits_microinteger required

Same as estimated_credits in micro-credits.

submitted_atstring date-time required

When the job was accepted.

started_atstring date-time

When the first item started. Absent until then.

finished_atstring date-time

When the job reached a terminal status. Cleared again by batchRetry.

cancel_requested_atstring date-time

When a cancel was accepted. The job may still be cancelling after this while in-flight items drain.

results_expire_atstring date-time

Results deadline: submitted_at + 72 h by default. After it, GET .../results answers 410. Download before this time.

error_codestring

Machine code of a job-level failure (for example the job was aborted). Absent otherwise.

error_messagestring

Human-readable job-level failure reason.

warningsstring[]

Decisions made on your behalf at submission — a clamped concurrency, or items not yet handed to the queue (do NOT resubmit in that case). Present only on the create response.

Example response

{
  "id": "01J9Z6T3W9E21T5TZARVJRVN5C",
  "project_id": "01J8X2M4C7Q1H5V9P3R6T8W0YB",
  "name": "catalog-2026-09-22",
  "status": "running",
  "status_url": "/v1/batch/01J9Z6T3W9E21T5TZARVJRVN5C",
  "results_url": "/v1/batch/01J9Z6T3W9E21T5TZARVJRVN5C/results",
  "params": {
    "js_render": true,
    "proxy_country": "us",
    "credit_budget_micro": 8000000
  },
  "total_items": 2,
  "concurrency": 20,
  "priority": 100,
  "max_attempts": 3,
  "estimated_credits": 8,
  "estimated_credits_micro": 8000000,
  "progress": {
    "total": 2,
    "completed": 1,
    "succeeded": 1,
    "failed": 0,
    "cancelled": 0,
    "skipped": 0,
    "remaining": 1,
    "percent_complete": 50,
    "credits_charged": 4,
    "credits_charged_micro": 4000000,
    "bytes": 326000
  },
  "submitted_at": "2026-09-22T06:41:12Z",
  "started_at": "2026-09-22T06:41:13Z",
  "results_expire_at": "2026-09-25T06:41:12Z"
}

Changes

No recorded changes to this operation across all 1 revision of this API.