Sign in
scraping

Scrape Bulk

Bulk scrape multiple URLs with checkpointing to S3

Requires permission: scrape:write

post/scrape/bulk

Request body

urlsstring[] required

List of URLs to scrape

crawl_idstring required

Unique identifier for this crawl

s3_bucketstring required

S3 bucket for checkpointing

max_workersinteger

Maximum concurrent workers

batch_sizeinteger

URLs per batch

debugboolean

Enable debug information

Example request

{
  "batch_size": 50,
  "crawl_id": "crawl-123",
  "engines": [
    {
      "engine_type": "scrapingbee"
    }
  ],
  "max_workers": 8,
  "s3_bucket": "my-scraping-bucket",
  "urls": [
    "https://example.com",
    "https://example.org"
  ]
}

Response

Successful Response

crawl_idstring required
total_urlsinteger required
total_pages_visitedinteger required
success_countinteger required
failed_countinteger required
cached_countinteger required
save_failed_countinteger required
timestampstring required
debug_frameobject[] nullable

Changes