---
title: "Start a crawl"
method: POST
path: "/v1/web/crawl"
tags: ["Crawl & batch"]
---

# Start a crawl

`POST /v1/web/crawl`

## Request

- Base URL: `https://api.hydrafetch.com`
- URL: `https://api.hydrafetch.com/v1/web/crawl`
- Auth: API key in header `X-API-Key`

Discover and scrape a whole site as one asynchronous job. Returns a crawl id immediately; poll it or register a webhook. One credit per page scraped.

## Request body

- CrawlRequestDto
  - `url` string, required: The site to start from. Must be http(s).
  - `limit` number: Maximum number of pages to scrape.
  - `maxDepth` number: How many links deep from the starting page to follow.
  - `includePaths` string[]: Only follow URLs whose path matches every one of these patterns.
  - `excludePaths` string[]: Skip URLs whose path matches any of these patterns.
  - `allowSubdomains` boolean: Also follow links into subdomains of the starting site. Default off.
  - `allowExternalLinks` boolean: Also follow links that lead off the starting site. Default off.
  - `ignoreQueryParameters` boolean: Treat URLs that differ only by query string as the same page. Default off.
  - `sitemap` 'skip' | 'include': Whether to seed discovery from the site's published page list. Default includes it.
  - `webhook` WebhookDto
    - `url` string, required: The URL to POST events to. Must be a public http(s) URL — private, loopback, and link-local addresses are rejected.
    - `events` string[]: Which events to receive. `page` fires as each page finishes; `completed` fires once when the job ends. Defaults to both.
    - `headers` object: Extra headers to send with each callback, e.g. for authentication.
    - `secret` string: A shared secret. When set, each callback carries an X-Hydrafetch-Signature header you can verify to prove the request came from us and is not a replay.
  - `scrapeOptions` ScrapeOptionsDto
    - `formats` string[]: What to return per page. Omit for Markdown only.
    - `onlyMainContent` boolean: Return only the main content, dropping boilerplate. Default true.
    - `includeTags` string[]: CSS selectors to keep.
    - `excludeTags` string[]: CSS selectors to strip before extraction.
    - `removeBase64Images` boolean: Strip inline base64 images. Default true.
    - `blockAds` boolean: Remove common ad and tracking elements. Default true.
    - `includeLinks` boolean: Keep inline links in the markdown. By default links are kept wherever the page text carries them. Set true to keep them on every page, or false for the densest prose.
    - `waitFor` number: Extra milliseconds to let the page settle before capture.
    - `timeout` number: Per-page time budget, in milliseconds.
    - `location` OptLocationDto
      - `country` string: ISO 3166 alpha-2 country to fetch as if from.
      - `languages` string[]: Preferred content languages, most-preferred first.
    - `headers` object: Extra request headers to send when fetching each page.
    - `preferStructure` boolean: Preserve document structure over prose density. Default off.
    - `maxAge` number: Serve a page from cache if younger than this many milliseconds.

## Response `201`

The crawl was accepted.

- CrawlAcceptedDto
  - `crawlId` string, required: Poll this crawl id for progress and results.
  - `status` 'queued', required: The job has been accepted.

## Other responses

- `400`: The request body or query failed validation.
- `401`: The X-API-Key header is missing, malformed, or names a key that no longer exists.
- `402`: The workspace has no credits left for this request. Nothing was charged and nothing was fetched.
- `403`: The key is valid but the workspace may not make this request.
- `404`: No such job, or the id belongs to another workspace.
- `422`: The target site did not respond, or we could not read that page. This is a fact about that URL rather than a fault on our side, and retrying it will not change the answer. When details.retryAfterSeconds is present, asking again before then returns this same answer. Nothing was charged.
- `429`: The workspace exceeded its per-minute request limit. Read the RateLimit headers, or Retry-After on this response, and retry after the window resets.
- `500`: Unexpected server error. Nothing was charged.
- `503`: A dependency needed for the requested formats is unavailable. Safe to retry.

---

[API](https://skmtc.dev/hydrafetch/apis/hydrafetch-api.md) · [All operations](https://skmtc.dev/hydrafetch/apis/hydrafetch-api/llms.txt) · [OpenAPI document](https://skmtc.dev/hydrafetch/apis/hydrafetch-api/revisions/b28322984b4b?raw)