scraping

Scrape Bulk

Bulk scrape multiple URLs with checkpointing

Requires permission: scrape:write

post/scrape/bulk

Request body

urlsstring[] required

List of URLs to scrape

crawl_idstring required

Unique identifier for this crawl

max_workersinteger

Maximum concurrent workers

batch_sizeinteger

URLs per batch

debugboolean

Enable debug information

Example request

{
  "batch_size": 50,
  "crawl_id": "crawl-123",
  "engines": [
    {
      "engine_type": "scrapingbee"
    }
  ],
  "max_workers": 8,
  "urls": [
    "https://example.com",
    "https://example.org"
  ]
}

Response

Successful Response

crawl_idstring required

total_urlsinteger required

total_pages_visitedinteger required

success_countinteger required

failed_countinteger required

cached_countinteger required

save_failed_countinteger required

timestampstring required

debug_frameobject[] nullable

Changes

Changed in 4 of the 16 revisions of this API.519

Feb 17, 202629225d7b3ea412See the full diff
- ●
  removed the request property s3_bucket
  request-property-removed
- ○
  added the optional property detail/items/ctx to the response with the 422 status
  response-optional-property-added
- ○
  added the optional property detail/items/input to the response with the 422 status
  response-optional-property-added
Sep 18, 2025019c4555bd7f1See the full diff
- ○
  added the new ASYNC_FLEET_STICKY enum value to the request property engines/items/engine_type
  request-property-enum-value-added
Aug 12, 2025707c0eb9cee112See the full diff
- ▲
  removed the enum value ASYNC_FLEET of the request property engines/items/engine_type
  request-property-enum-value-removed
- ○
  added the new FLEET_ASYNC enum value to the request property engines/items/engine_type
  request-property-enum-value-added
- ○
  added the new FLEET_WORKFLOW enum value to the request property engines/items/engine_type
  request-property-enum-value-added
Aug 5, 20254eb68ae8508044See the full diff
- ▲
  removed the enum value async_fleet of the request property engines/items/engine_type
  request-property-enum-value-removed
- ▲
  removed the enum value fleet of the request property engines/items/engine_type
  request-property-enum-value-removed
- ▲
  removed the enum value scrapingbee of the request property engines/items/engine_type
  request-property-enum-value-removed
- ▲
  removed the enum value zenrows of the request property engines/items/engine_type
  request-property-enum-value-removed
- ○
  added the new ASYNC_FLEET enum value to the request property engines/items/engine_type
  request-property-enum-value-added
- ○
  added the new FLEET enum value to the request property engines/items/engine_type
  request-property-enum-value-added
- ○
  added the new SCRAPINGBEE enum value to the request property engines/items/engine_type
  request-property-enum-value-added
- ○
  added the new ZENROWS enum value to the request property engines/items/engine_type
  request-property-enum-value-added

Full history