Crawl & batch

Start a batch

Scrape an explicit list of URLs as one asynchronous job. Returns a batch id immediately; poll it or register a webhook. One credit per URL scraped.

post/v1/web/batch

Request

  • Base URL: https://api.hydrafetch.com
  • URL: https://api.hydrafetch.com/v1/web/batch
  • Auth: API key in header X-API-Key

Request body

urlsstring[] required

The explicit list of URLs to scrape. Each must be http(s).

Example request

{
  "urls": [
    "https://example.com/a",
    "https://example.com/b"
  ],
  "webhook": {
    "url": "https://your-app.com/hooks/hydrafetch",
    "events": [
      "page",
      "completed"
    ],
    "headers": {
      "Authorization": "Bearer your-token"
    },
    "secret": "whsec_a_long_random_string"
  },
  "scrapeOptions": {
    "formats": [
      "markdown"
    ],
    "location": {
      "country": "us",
      "languages": [
        "en-US",
        "en"
      ]
    }
  }
}

Response

The batch was accepted.

batchIdstring required

Poll this batch id for progress and results.

status'queued' required

The job has been accepted.

Example response

{
  "batchId": "019f3c09-6fae-740f-9257-10c2b6af7f43",
  "status": "queued"
}

Changes

No recorded changes to this operation across all 1 revision of this API.