Scrape

Scrape a URL

Fetch a URL through Kadoa. Defaults to smart routing (auto fetch mode and proxy), can return page content as markdown for LLM consumption, and offers a stealth proxy tier for sites that block bots.

post/v4/scrape

Request

  • Base URL: https://api.kadoa.com
  • URL: https://api.kadoa.com/v4/scrape
  • Auth: API key in header x-api-key

Request body

urlstring uri required
method'GET' | 'POST' | 'PUT' | 'DELETE' | 'PATCH'

HTTP method. Default: GET

rawBodystring

Exact request body text to forward to the upstream URL. Use for endpoints that reject wrapper objects or require a specific JSON/form payload string.

timeoutinteger

Per-attempt timeout in milliseconds

overallTimeoutinteger

Wall-clock deadline for the complete scrape in milliseconds. Default and maximum: 240000

captureResponsesstring[]

Substrings matched against network response URLs (browser workers only). The worker hooks every response the page makes and returns the body of each whose URL contains any of these substrings, in capturedResponses. Use this to read data that arrives via XHR/fetch and is NOT in the DOM — SPA dashboards (Tableau VizQL, Power BI) where values come back in a JSON/command response. Triggers a browser worker; pair with browserActions (clicks/waits) to drive the requests you want captured.

waitUntil'load' | 'domcontentloaded' | 'networkidle' | 'commit'

Browser-worker navigation gate. Default (omitted) is domcontentloaded + a bounded best-effort networkidle settle, which returns content even on heavy pages that never go fully idle (ASP.NET, SignalR, ad/telemetry-heavy). Set networkidle to block the whole navigation on full idle (old behavior — only when the page reliably goes idle and you need every lazy request). No-op for non-browser workers.

settleMsinteger

Browser-worker cap (ms) for the best-effort networkidle settle after domcontentloaded. Default 15000. Raise for slow client-rendered pages that need more time for lazy content; lower to return faster. Ignored when waitUntil: networkidle and for non-browser workers.

devicestring

Device to emulate, by descriptor name (e.g. 'iPad', 'iPhone 13'). Honored by workers that support device emulation; ignored by the rest. Some sites only serve content for a given device profile.

fetchMode'http' | 'browser' | 'auto'

Fetch mode: http for fast requests, browser for JavaScript rendering and browser actions, or auto for smart routing that starts cheap and escalates when blocked. Default: auto

proxy'auto' | 'direct' | 'dc' | 'isp' | 'residential' | 'stealth'

Proxy tier: auto (smart selection), direct (Squid server egress without location guarantees), dc (datacenter), isp, residential, or stealth to route through the anti-bot providers for sites that block bots. Stealth providers use their own egress and pricing. Default: auto

format'html' | 'markdown'

Content format for HTML pages. markdown converts the page to markdown with navigation, header, and footer stripped — the compact shape to feed an LLM. Non-HTML responses (JSON, binary) and HTML pages over 2 MB are returned unchanged; check the response contentType to see which format was applied. Default: html

Response

Scrape response with content and execution metadata

retryableboolean

Whether repeating this scrape is safe and useful; absence does not authorize retry

retryAfterMsinteger

Minimum delay before another scrape attempt, in milliseconds

successboolean required

Whether the scrape succeeded

contentTypestring

MIME type of the response (e.g., text/html, application/json, image/png)

contentstring

Response content (raw text for html/json, base64 for binary)

isBase64boolean

Whether content is base64-encoded (true for binary content types like PDFs, images)

screenshotsstring[]

URLs of captured screenshots

statusCodenumber

HTTP status code from the response

headersobject

Response headers as key-value pairs

finalUrlstring

Final URL after any redirects

Changes

No recorded changes to this operation across all 1 revision of this API.