Crawl a site asynchronously

post/v2/crawl

Request

  • Base URL: https://api.deepsfetch.com
  • URL: https://api.deepsfetch.com/v2/crawl
  • Auth: HTTP bearer

Headers

Idempotency-Keystring

Request body

asyncboolean
device'desktop' | 'mobile' | 'tablet'

Legacy common option. Not forwarded by the HTML adapter or recognized by the current browser runtime. Do not rely on it to customize execution.

urlstring uri required
limitinteger

Effective adapter maximum: 500. Higher values fail with provider_limit_exceeded.

includestring[]
excludestring[]
maxDepthinteger

Forwarded to the map/crawl adapter.

includeSubdomainsboolean
respectRobotsboolean
useSitemapboolean

Response

Accepted

idstring required
kind'scrape' | 'extract' | 'search' | 'map' | 'crawl' | 'interact' | 'watch' required
status'queued' | 'running' | 'completed' | 'failed' | 'cancelled' required
resultobject nullable

Provider-specific output. V1 adapter: provider, operation, providerJobId, status, statusUrl, result, meta. Inspect inner status for partial completion; browser shape differs.

createdAtstring date-time required
startedAtstring nullable
completedAtstring nullable
cancelledAtstring nullable

Changes

No recorded changes to this operation across all 1 revision of this API.