Sign in
downloading

Download Bulk

Bulk download multiple files with checkpointing to S3

Requires permission: download:write

post/download/bulk

Request body

urlsstring[] required

List of URLs to download

crawl_idstring required

Unique identifier for this crawl

s3_bucketstring required

S3 bucket for storage and checkpoints

downloader_type'async_fleet'

Available downloader types

max_workersinteger

Maximum concurrent workers

batch_sizeinteger

URLs per batch

wait_until'commit' | 'domcontentloaded' | 'load' | 'networkidle'

When to consider download complete

debugboolean

Enable debug information

Example request

{
  "crawl_id": "download-123",
  "downloader_type": "async_fleet",
  "max_workers": 8,
  "s3_bucket": "my-downloads",
  "urls": [
    "https://example.com/file1.pdf",
    "https://example.com/file2.pdf"
  ]
}

Response

Successful Response

crawl_idstring required
total_urlsinteger required
total_downloads_attemptedinteger required
success_countinteger required
failed_countinteger required
cached_countinteger required
save_failed_countinteger required
timestampstring required
debug_frameobject[] nullable

Changes