Web

Crawl web pages

Crawl a small set of web pages synchronously.

get/v1/web/crawl

Request

  • Base URL: https://api.socialfetch.dev
  • URL: https://api.socialfetch.dev/v1/web/crawl
  • Auth: one of:
    • API key in header x-api-key
    • none

Query parameters

urlstring[] required

URLs to crawl. Repeat the url query parameter for multiple pages (max 5).

scanFullPageboolean

When true, scroll the page to load dynamically appended content (infinite scroll). Default false.

waitForstring

Wait for a CSS selector before extraction. Must be prefixed with "css:" (e.g. css:main). JavaScript wait conditions are not supported.

Response

Crawl results.

Changes

No recorded changes to this operation across all 1 revision of this API.