Sign in
Batch

Submit a batch

Scrape a list of URLs or crawl a site asynchronously. Send an Idempotency-Key header to make retries safe: resubmitting with the same key returns the batch the first attempt created instead of creating and charging a second one.

post/batch/submit

Headers

Idempotency-Keystring

Any string unique to this submission, at most 200 characters. Retries with the same key return the original batch; reusing a key with a different body fails with IDEMPOTENCY_KEY_CONFLICT.

Any string unique to this submission, at most 200 characters. Retries with the same key return the original batch; reusing a key with a different body fails with IDEMPOTENCY_KEY_CONFLICT.

Request body

webhookUrlstring

URL notified when the batch finishes.

tagsstring[]

Tags stored on the batch. Filter the batch list by them later.

Example request

{
  "input": {
    "data": {
      "urls": [
        {
          "url": "https://example.com/products/anvil",
          "itemId": "sku-1"
        }
      ],
      "options": {
        "country": "de"
      }
    }
  },
  "tags": [
    "docs",
    "competitor"
  ]
}

Response

Batch created

idstring required

Batch ID used to retrieve or cancel the job.

status'queued' | 'running' | 'cancelling' | 'completed' | 'cancelled' | 'failed' required

Current state. completed, cancelled, and failed are final.

mode'scrape' | 'crawl' required

How pages are selected.

type'markdown' | 'html' required

Output format.

tagsstring[] required

Tags stored on the batch at submission.

webhook_secretstring

Webhook signing secret. Also returned by GET /batch/{batch_id}.

Example response

{
  "id": "batch_9f2c8a",
  "tags": [
    "docs"
  ],
  "input": {
    "submitted": 25000,
    "accepted": 24817,
    "duplicates": 183
  },
  "progress": {
    "succeeded": 18091,
    "failed": 311,
    "pending": 6415
  },
  "credits": {
    "estimated": 24817,
    "charged": 18091
  },
  "errors": [
    {
      "code": "WEBSITE_ACCESS_ERROR",
      "count": 204
    }
  ],
  "invalid_urls": [
    {
      "reason": "Must be a public http:// or https:// URL"
    }
  ]
}

Changes