crawls

Start a crawl of a site

Start a full crawl of a site. It answers at once with the crawl's handle; a crawl takes minutes to hours, so poll crawls_get with its id. One crawl runs per site: a call while one is running joins it (crawl.deduplicated: true) and starts nothing. Pass the same idempotency_key on a retry to get the same crawl; without one, every call is a new request. An agent may start 5 crawls per site per UTC day, failed and cancelled ones included; a join never counts. Over that, RATE_LIMITED: ask your human to start it from Frontshelf. Run it with dry_run first to see whether it would start or join and how many crawls today leaves. Do not crawl to check a fix: ask the verifier with verifications_start instead, which is scoped and cheaper.

post/v1/sites/{site}/crawls

Request

  • Base URL: https://frontshelf.io/api
  • URL: https://frontshelf.io/api/v1/sites/{site}/crawls
  • Auth: HTTP bearer

Path parameters

sitestring required

Site ID (site_…), origin (https://example.com), or bare domain, resolved server-side; the response carries the canonical ID.

Headers

Idempotency-Keystring

Any unique string per intended change; a retry with the same key answers the first call's result.

Request body

reasonstring

A note recorded on the run.

mode'spider' | 'list' | 'serp'

spider follows links from the origin; list crawls exactly a list of URLs; serp fetches nothing and judges a list's titles and descriptions. Absent: the site's configured mode (spider by default).

list_idstring

A URL list id (ul_...) for a list or serp run; absent: the site's configured list sources.

dry_runboolean

true: check the call and say what it would do, without doing it.

confirmboolean

true: go ahead with a change that asks for confirmation.

Response

Start a crawl of a site

summarystring required

The answer in one or two sentences, with the numbers it rests on.

warningsstring[] required

What limits this answer (a cap, a read that was cut short). Empty when nothing does.

Changes

No recorded changes to this operation across all 1 revision of this API.