crawl

Start an async crawl job

post/v1/crawl

Request

  • Base URL: https://api.fastcrw.com
  • URL: https://api.fastcrw.com/v1/crawl
  • Auth: one of:
    • HTTP bearer
    • none

Request body

urlstring uri required
maxPagesinteger
maxDepthinteger
headersobject

Custom HTTP request headers applied to every page the crawl fetches. On a browser render they apply to every request the page makes, subresources included, so avoid cross-origin-sensitive credentials here.

formatsCrawlOutputFormat[]
onlyMainContentboolean
jsonSchemaobject
renderJsboolean nullable
waitForinteger
renderer'auto' | 'lightpanda' | 'chrome' | 'chrome_proxy' | 'playwright' | 'camoufox' | 'impersonated-http'
countrystring

2-letter ISO 3166-1 alpha-2 proxy egress country.

proxyListstring[]
proxyRotation'round_robin' | 'random' | 'sticky_per_host'

Response

Crawl job accepted; poll with /v1/crawl/{id}

successboolean required
idstring required
urlstring uri

URL to poll with GET /v1/crawl/{id}

Changes

No recorded changes to this operation across all 1 revision of this API.