Scrape a web page
Fetches a public web page and returns clean content, metadata, and optional media in the unified WebPage schema.
Request
- Base URL: https://www.socialcrawl.dev/v1
- URL: https://www.socialcrawl.dev/v1/web/scrape
- Auth: API key in header x-api-key
Query parameters
Public URL to fetch.
Comma-separated output formats such as markdown,screenshot.
Strip nav, headers, footers and sidebars and keep the article body. Default true; send false when you need the whole page, including the chrome. A markdown-only scrape of a site the primary scraper cannot fetch is served by a second extractor, which returns the whole page.
Milliseconds to wait after load before capturing, for pages that render their content in JavaScript. Leave unset unless the page comes back empty or half-built.
Render in a mobile viewport with a mobile user agent. Use it when the site serves a different layout to phones.
Hard cap on the page load in milliseconds, 1 to 30000. A page that exceeds it fails and is refunded rather than returning a partial capture.
Accept an upstream-cached copy up to this many milliseconds old, which is faster and still one credit. Default 172800000 (48 hours), or 0 (a live fetch) with proxy=auto or proxy=enhanced; send 0 to force a live fetch. A cached copy of a page that failed (HTTP 400 or more) is never served: it is fetched again, live.
ISO 3166-1 alpha-2 country code to fetch from, e.g. 'us' or 'de'. Use it for geo-varying pages such as pricing or availability.
When a screenshot format is requested, capture the entire scrollable page instead of the visible viewport.
CSV of CSS selectors or HTML tags to keep, e.g. 'article,main'. Everything outside them is dropped. Use it to pin extraction to a known container.
CSV of CSS selectors or HTML tags to drop, e.g. 'nav,footer,.cookie-banner'. Applied after include_tags.
Proxy tier: basic (the default), auto (a plain request first, then a stronger one if the site blocks it), or enhanced. Use auto when a plain scrape returns a block or CAPTCHA page. auto and enhanced read the page live unless you set max_age.
Parse the target as a PDF and return its extracted text. Send it when the URL points at a PDF rather than an HTML page.
Block ad and tracker requests, which is faster and quieter. Default true.
Drop inline base64-encoded images from the returned content instead of carrying them in the payload. Default true.
Headers
Send no-cache to bypass the response cache and force a live fetch. Billed at the normal endpoint cost; the fresh result is written back to cache for the next caller. Only the no-cache directive triggers this. See the Response Schema guide for details.
Optional UUID that makes the request safely retriable. The key is reserved atomically per account before billing. A concurrent retry receives 409 IDEMPOTENCY_IN_PROGRESS with a Retry-After of 1 second while the original is young, growing to a quarter of the time it has been running (at most 30); a crashed reservation expires after 5 minutes. A replay keeps the cached payload immutable except for billing metadata: credits_used becomes 0, idempotent_replay becomes true, and credits_remaining is refreshed to the current balance. A known current balance appears in both the body and X-Credits-Remaining header; no balance row resolves to 0. On a transient lookup failure, body credits_remaining is null and X-Credits-Remaining is omitted. Completed outcomes expire after 24 hours. If the claim store is unavailable, the request fails closed before billing. GET endpoints do not combine idempotent replay with Server-Sent Events: use Accept: application/json when sending this header. The unsupported key-plus-SSE combination is rejected before billing.
Response
Successful response
Example response
{
"success": true,
"platform": "web",
"endpoint": "/v1/web/scrape",
"credits_used": 1,
"credits_remaining": 4999,
"request_id": "req-abc123",
"idempotent_replay": true,
"meta": {
"hint": {
"code": "search_multi",
"message": "You searched \"matcha\" on 3 platforms one call at a time; /v1/search/multi returns each platform's own search rows in one call at the same price per platform, and a platform with no rows is not charged.",
"method": "GET",
"path": "/v1/search/multi?query=matcha&platforms=instagram,tiktok,youtube"
}
}
}Changes
No recorded changes to this operation across all 1 revision of this API.