---
title: "Start a crawl of a site"
method: POST
path: "/v1/sites/{site}/crawls"
tags: ["crawls"]
---

# Start a crawl of a site

`POST /v1/sites/{site}/crawls`

## Request

- Base URL: `https://frontshelf.io/api`
- URL: `https://frontshelf.io/api/v1/sites/{site}/crawls`
- Auth: HTTP bearer

Start a full crawl of a site. It answers at once with the crawl's handle; a crawl takes minutes to hours, so poll crawls_get with its id. One crawl runs per site: a call while one is running joins it (`crawl.deduplicated: true`) and starts nothing. Pass the same idempotency_key on a retry to get the same crawl; without one, every call is a new request. An agent may start 5 crawls per site per UTC day, failed and cancelled ones included; a join never counts. Over that, RATE_LIMITED: ask your human to start it from Frontshelf. Run it with dry_run first to see whether it would start or join and how many crawls today leaves. Do not crawl to check a fix: ask the verifier with verifications_start instead, which is scoped and cheaper.

## Path parameters

- `site` string, required: Site ID (site_…), origin (https://example.com), or bare domain, resolved server-side; the response carries the canonical ID.

## Headers

- `Idempotency-Key` string: Any unique string per intended change; a retry with the same key answers the first call's result.

## Request body

- object
  - `reason` string: A note recorded on the run.
  - `mode` 'spider' | 'list' | 'serp': `spider` follows links from the origin; `list` crawls exactly a list of URLs; `serp` fetches nothing and judges a list's titles and descriptions. Absent: the site's configured mode (spider by default).
  - `list_id` string: A URL list id (ul_...) for a list or serp run; absent: the site's configured list sources.
  - `dry_run` boolean: true: check the call and say what it would do, without doing it.
  - `confirm` boolean: true: go ahead with a change that asks for confirmation.

## Response `202`

Start a crawl of a site

- object
  - `summary` string, required: The answer in one or two sentences, with the numbers it rests on.
  - `warnings` string[], required: What limits this answer (a cap, a read that was cut short). Empty when nothing does.
  - `next_actions` object[], required: What to call next, best first. Follow them.
    - `capability` string, required: What to call next: a capability.
    - `arguments` object, required: Its arguments, complete; pass them as they are.
    - `instruction` string, required: What to do, written to you.
  - `crawl` object, required: The crawl this call started, joined or replayed.
    - `id` string, required: The crawl id crawls_get takes.
    - `site_id` string, required: The canonical id of the site crawled.
    - `crawl_seq` integer, required: The site's crawl number.
    - `status` 'queued' | 'leased' | 'running' | 'finalizing' | 'complete' | 'complete_with_exclusions' | 'partial_budget' | 'partial_blocked' | 'failed_infrastructure' | 'cancelled', required: `queued` until a crawler picks it up; on a joined or replayed crawl, wherever that crawl is.
    - `mode` 'spider' | 'list' | 'serp', required: The run kind the crawl was started with.
    - `list_id` string, nullable, required: The URL list a list or serp run reads; null for a spider run or the site's own list sources.
    - `deduplicated` boolean, required: True: this call joined the site's running crawl, or replayed a key that already started one; nothing new started.
  - `estimated_usage` object, required: What the crawl may use of the site's plan.
    - `max_pages` integer, required: The site's page limit: an upper bound on what the crawl reads, not a prediction.

## Other responses

- `400`: VALIDATION_FAILED
- `401`: UNAUTHENTICATED
- `402`: BILLING_REQUIRED
- `403`: ORGANIZATION_FROZEN, SCOPE_DENIED
- `404`: SITE_NOT_FOUND, NOT_FOUND
- `409`: AMBIGUOUS_SITE, IDEMPOTENCY_CONFLICT, AGENT_STOPPED
- `428`: CONFIRMATION_REQUIRED
- `429`: RATE_LIMITED
- `503`: PLATFORM_READ_ONLY, DEPENDENCY_UNAVAILABLE

---

[API](https://skmtc.dev/frontshelf/apis/frontshelf-api.md) · [All operations](https://skmtc.dev/frontshelf/apis/frontshelf-api/llms.txt) · [OpenAPI document](https://skmtc.dev/frontshelf/apis/frontshelf-api/revisions/ce3602d26c7c?raw)