Sign in
downloading

Download Bulk

Bulk download multiple files with checkpointing

Requires permission: download:write

post/download/bulk

Request body

urlsstring[] required

List of URLs to download

crawl_idstring required

Unique identifier for this crawl

downloader_type'FLEET_ASYNC'

Available downloader types

max_workersinteger

Maximum concurrent workers

batch_sizeinteger

URLs per batch

wait_until'domcontentloaded' | 'load' | 'networkidle' | 'commit'

When to consider page load complete for web scraping operations

debugboolean

Enable debug information

Example request

{
  "crawl_id": "download-123",
  "downloader_type": "FLEET_ASYNC",
  "max_workers": 8,
  "urls": [
    "https://example.com/file1.pdf",
    "https://example.com/file2.pdf"
  ]
}

Response

Successful Response

crawl_idstring required
total_urlsinteger required
total_downloads_attemptedinteger required
success_countinteger required
failed_countinteger required
cached_countinteger required
save_failed_countinteger required
timestampstring required
debug_frameobject[] nullable

Changes

Changed in 2 of the 16 revisions of this API.114

  • 29225d7b3ea412See the full diff
    • ●

      removed the request property s3_bucket

      request-property-removed

    • ○

      added the optional property detail/items/ctx to the response with the 422 status

      response-optional-property-added

    • ○

      added the optional property detail/items/input to the response with the 422 status

      response-optional-property-added

  • 707c0eb9cee112See the full diff
    • ▲

      removed the enum value async_fleet of the request property downloader_type

      request-property-enum-value-removed

    • ○

      the downloader_type request property default value changed from async_fleet to FLEET_ASYNC

      request-property-default-value-changed

    • ○

      added the new FLEET_ASYNC enum value to the request property downloader_type

      request-property-enum-value-added