--- title: "Scrape a URL" method: POST path: "/v1/scrape" tags: ["Scrape"] --- # Scrape a URL `POST /v1/scrape` ## Request - Base URL: `https://api.spicrawl.com` - URL: `https://api.spicrawl.com/v1/scrape` - Auth: HTTP bearer Fetch one URL synchronously and return it as html, markdown, text, a PDF, or a JSON envelope with extracted data. How a request is rendered (cheapest first). You do not choose: Spicrawl picks the cheapest path that can do what the request asks. - Default (plain fetch, 1 credit): plain HTTP, no JavaScript, presenting a current browser TLS/HTTP2 fingerprint (`impersonate`, on by default and free; `impersonate=false` turns it off). - `js_render=true` (JavaScript rendering, 3 credits): content is built by JavaScript, or you need wait/wait_for/block_resources/network_capture. - Screenshots, `response_format=pdf` and `actions` use full-browser rendering (8 credits). `stealth=true` (full-browser rendering hardened against bot detection, 25 credits) is coming soon. - `mode=auto`: unsure - escalates through the available rendering tiers, bills only the tier that worked. - Managed proxy pools (`premium_proxy`) are coming soon; during the beta pass your own proxy with `proxy`. - The `engine` and `headless` fields are deprecated and ignored. Browser-only flags on the fetch tier are a 400, never silently ignored. Billing: failures cost 0 (errors, bot challenges, timeouts, target statuses other than 200/404/410 unless listed in allowed_status_codes, undelivered screenshots). A cache hit is billed at the same price as the fetch that stored it: the cache saves the fetch (latency and load on the target), not credits. Like any request, a hit is refused with 402 when the monthly allowance cannot cover its price. Check X-Credits-Charged. Bot challenges: a challenge page attributed to a named vendor (Cloudflare, DataDome, PerimeterX, Akamai) is never returned as a success. A fetch through a managed pool exit that meets one is retried once on a different exit, free (GET/HEAD/OPTIONS only, not with `sticky_key` or `session_id`, never on your own `proxy`). When you bring your own `proxy` (residential works best) and send no `mode=auto`, a request whose page is still a challenge is escalated automatically to stronger full-browser rendering on the same exit, which works through the interstitial and returns the real page; there is no flag to set. This is how Cloudflare-protected pages are supported. Automatic handling on managed premium exits is coming soon. Success is not guaranteed for every page. Only the tier that returned the page is billed (up to 25 credits, or the first tier's price), and 0 if every tier was challenged (502 ERR::UPSTREAM::CHALLENGE). On the default shared pool a challenged page may return that error; retry with your own residential `proxy`. A transport error, a refusal with no vendor marker (such as a bare 403) or a non-billable status such as 404 does not escalate. Escalation is skipped for a non-GET method, response_format=pdf, a screenshot, actions, a `proxy` that is not http://, or a max_cost below the escalated price; an allowance that covers the first attempt but not the escalation runs the request without it. X-Engine reports the rendering tier that served, each abandoned tier adds an `X-Warning: ESCALATED`, and diagnostics.attempts lists every attempt (in the JSON envelope on a successful escalation). Batch items never escalate. Status: the HTTP status is the platform's. A blocked or erroring site is usually still HTTP 200 - read envelope `status` or X-Target-Status for the site's answer. Errors are application/problem+json; retry only when `retryable` is true. Not idempotent: there is no idempotency key, and each call bills again, whether it performs a new scrape or is served from cache. GET and POST accept the same parameters and behave identically. ## Request body - ScrapeRequest (unresolved $ref) ## Response `200` Scrape completed. The HTTP status is the platform's: the site's status is envelope `status` / X-Target-Status, and a non-billable target status (e.g. 403, 500) still returns 200 with the site's body at 0 credits. With original_status=true this status line carries the target's status instead. - ScrapeEnvelope: Returned for response_format=json, or whenever extract/autoparse/links/ai_extract/network_capture/screenshot is set. - `url` string, required: The requested URL. - `final_url` string, required: URL after redirects; may be empty. - `status` integer, required: The TARGET's HTTP status (not this response's). Anything other than 200/404/410 or allowed_status_codes was charged 0. - `content` string, required: The page in the requested rendering: html (cleaned html if extraction produced one), markdown or text. - `headers` object, nullable, required: The target's response headers. - `truncated` boolean, required: The target body exceeded the size cap and was cut. - `credits` integer, required: Credits charged for this request (same as X-Credits-Charged). - `engine` 'fetch' | 'js' | 'browser': The rendering tier used to produce the page. `fetch` is HTTP only, `js` is JavaScript rendering, `browser` is full-browser rendering. - `proxy_source` 'pool' | 'custom' | 'direct', required - `warnings` string[], required: Human-readable warnings (messages only; codes are on X-Warning). - `data` object: Output of extract, autoparse or ai_extract. - `empty_fields` string[]: Extraction rules that matched nothing - the usual sign a selector broke. - `links` string[]: Present (possibly empty) when links=true. - `screenshots` ScrapeScreenshot[]: Delivered images only; undelivered ones are reported in warnings and charged 0. - `label` string, required: `final` for the top-level screenshot, or the owning action's label. - `encoding` 'base64', required - `data` string, byte, required: Image bytes. - `size_bytes` integer, required: Decoded length of `data`. - `format` 'png' | 'jpeg' | 'webp' - `width` integer: Omitted when not measured. - `height` integer: Omitted when not measured. - `network` ScrapeNetworkResponse[]: Captured responses when network_capture was set. - `url` string, required - `status` integer, required - `resource_type` string, required - `content_type` string - `duration_ms` integer - `from_cache` boolean - `body_truncated` boolean: Body was cut at max_body_bytes; truncated JSON is delivered as base64. - `body` unknown - `body_base64` string, byte: Any other body. - `encoding` 'base64': Present with body_base64. - `diagnostics` ScrapeDiagnostics: Present on a successful mode=auto escalation: which rungs were tried and why they were abandoned. - `failed_at` string - `timeline` object[] - `stage` string, required - `ms` integer, required - `ok` boolean, required - `detail` string - `attempts` object[] - `n` integer, required - `engine` 'fetch' | 'js' | 'browser' - `proxy` string - `outcome` string, required - `ms` integer, required - `error_code` string - `budget_ms` integer - `elapsed_ms` integer - `engine` 'fetch' | 'js' | 'browser' - `proxy` object - `source` string - `endpoint` string - `country` string - `hint` string ## Other responses - `400`: The request is invalid. Not retryable; fix the request using `code` and `diagnostics.hint`. - `401`: Missing, invalid, revoked or expired key. Not retryable with the same key. - `402`: `ERR::LIMIT::QUOTA_EXCEEDED`: the organization is out of credits. Not retryable until credits are added. - `403`: The key lacks the scope this route requires (`ERR::AUTH::INSUFFICIENT_SCOPE`), or the action is not permitted. - `404`: No such resource for this key's organization. - `410`: The resource existed but is past its retention window or was purged. Not retryable. - `413`: The request body exceeds the size limit. - `429`: Rate, concurrency or live-session limit reached. Retryable after `Retry-After`. - `500`: Internal error. Retryable when `retryable` is true. - `502`: The target, the proxy or the render engine failed. Check `code` and `target_status`; most are retryable and cost 0 credits. - `503`: No proxy exit or engine capacity was available. Retryable after `Retry-After`; 0 credits. - `504`: The target did not answer in time (`ERR::UPSTREAM::TIMEOUT`). Retryable; 0 credits. See `diagnostics.hint`. --- [API](https://skmtc.dev/spicrawl/apis/spicrawl-api.md) · [All operations](https://skmtc.dev/spicrawl/apis/spicrawl-api/llms.txt) · [OpenAPI document](https://skmtc.dev/spicrawl/apis/spicrawl-api/revisions/c4b4c6f97219?raw)