---
title: "Start an async crawl of one site (1 credit per page read; `limit` credits reserved up front, the rest refunded)"
method: POST
path: "/v1/crawl"
---

# Start an async crawl of one site (1 credit per page read; \`limit\` credits reserved up front, the rest refunded)

`POST /v1/crawl`

## Request

- Base URL: `https://api.serpkite.com`
- URL: `https://api.serpkite.com/v1/crawl`
- Auth: HTTP bearer

From `url` and (unless `sitemap: skip`) the site's sitemaps, same host (optionally
subdomains), up to `limit` pages and `max_depth` link hops, filtered by
`include_paths` / `exclude_paths` (regular expressions matched against the URL
path). Breadth first, or best first with `query` (pages whose URL and link text
match it are read first). Each host's robots.txt is honoured (RFC 9309: rules,
`Crawl-delay`; a robots.txt that answers 5xx stops the crawl at the start page).
URLs are canonicalised (fragments and tracking parameters dropped, `www.`,
trailing slashes and redirects folded) so each page is read once. Each page comes
back as Markdown (or text) with its metadata. A crawl runs for at most 30 minutes
and returns up to 16 MB of pages; `result.stats.stopped` says why it ended.

Returns `202` with a task; poll `GET /v1/crawl/{id}` or wait for the signed
`crawl.completed` webhook. Pages read cost 1 credit each (0.5 from cache); a crawl
that read nothing fails (`no_pages`) and is refunded. At most 20 crawls per account
may be queued or running at once (`429`); a crawl interrupted by a deploy restarts
from the beginning.

## Request body

- CrawlRequest
  - `url` string, uri, required: Start page (always read: path filters and robots.txt apply to the pages found from it)
  - `limit` integer: Maximum pages read (and credits reserved)
  - `max_depth` integer: Link hops from url (sitemap pages count as 1 hop; 0 reads only url)
  - `include_paths` string[]: Regular expressions matched against the URL path; follow only paths matching one
  - `exclude_paths` string[]: Regular expressions matched against the URL path; never follow paths matching one
  - `include_subdomains` boolean
  - `sitemap` 'include' | 'only' | 'skip': `include` also seeds the crawl with the site's sitemap URLs (after the start page's links), `only` reads the start page and sitemap URLs without following links, `skip` follows links only
  - `query` string: Read the pages most relevant to these words first (best-first crawl)
  - `ignore_query_parameters` boolean: Treat URLs that differ only in their query string as one page
  - `format` 'markdown' | 'text'
  - `include_links` boolean: Return each page's outbound links
  - `max_tokens` integer: Trim each page to this many tokens
  - `max_age` integer: Accept cached pages up to this many seconds old (0.5 credit each)
  - `webhook_url` string, uri: Receives the signed `crawl.completed` event. Defaults to the account webhook

## Response `202`

Queued. Poll `poll_url` or wait for the webhook

- TaskCreated
  - `id` string, uuid, required
  - `kind` 'crawl', required
  - `status` 'queued' | 'running' | 'completed' | 'failed' | 'canceled', required
  - `created_at` string, date-time, required
  - `poll_url` string, required
  - `credits_reserved` number, required
  - `webhook_url` string, nullable

## Other responses

- `400`: Error
- `401`: Error
- `402`: Error
- `403`: Error
- `429`: Error
- `503`: Temporarily unavailable, not billed, safe to retry after `Retry-After`. Codes: `upstream_error` (the upstream page couldn't be fetched or parsed), `upstream_blocked` (the upstream is rate limiting us), `upstream_timeout` (the search took too long), `unavailable` (endpoint or dependency down). The API never sends 502 or 504.

---

[API](https://skmtc.dev/serpkite/apis/serpkite-serp-api.md) · [All operations](https://skmtc.dev/serpkite/apis/serpkite-serp-api/llms.txt) · [OpenAPI document](https://skmtc.dev/serpkite/apis/serpkite-serp-api/revisions/f2c6dd05cfc5?raw)