Crawl & batch

Start a crawl

Discover and scrape a whole site as one asynchronous job. Returns a crawl id immediately; poll it or register a webhook. One credit per page scraped.

post/v1/web/crawl

Request

  • Base URL: https://api.hydrafetch.com
  • URL: https://api.hydrafetch.com/v1/web/crawl
  • Auth: API key in header X-API-Key

Request body

urlstring required

The site to start from. Must be http(s).

limitnumber

Maximum number of pages to scrape.

maxDepthnumber

How many links deep from the starting page to follow.

includePathsstring[]

Only follow URLs whose path matches every one of these patterns.

excludePathsstring[]

Skip URLs whose path matches any of these patterns.

allowSubdomainsboolean

Also follow links into subdomains of the starting site. Default off.

allowExternalLinksboolean

Also follow links that lead off the starting site. Default off.

ignoreQueryParametersboolean

Treat URLs that differ only by query string as the same page. Default off.

sitemap'skip' | 'include'

Whether to seed discovery from the site's published page list. Default includes it.

Example request

{
  "url": "https://example.com",
  "limit": 100,
  "maxDepth": 2,
  "includePaths": [
    "^/blog/.*"
  ],
  "excludePaths": [
    "^/tag/.*"
  ],
  "sitemap": "include",
  "webhook": {
    "url": "https://your-app.com/hooks/hydrafetch",
    "events": [
      "page",
      "completed"
    ],
    "headers": {
      "Authorization": "Bearer your-token"
    },
    "secret": "whsec_a_long_random_string"
  },
  "scrapeOptions": {
    "formats": [
      "markdown"
    ],
    "location": {
      "country": "us",
      "languages": [
        "en-US",
        "en"
      ]
    }
  }
}

Response

The crawl was accepted.

crawlIdstring required

Poll this crawl id for progress and results.

status'queued' required

The job has been accepted.

Example response

{
  "crawlId": "019f3c09-6fae-740f-9257-10c2b6af7f43",
  "status": "queued"
}

Changes

No recorded changes to this operation across all 1 revision of this API.