---
title: "Scrape a URL"
method: POST
path: "/v4/scrape"
tags: ["Scrape"]
---

# Scrape a URL

`POST /v4/scrape`

## Request

- Base URL: `https://api.kadoa.com`
- URL: `https://api.kadoa.com/v4/scrape`
- Auth: API key in header `x-api-key`

Fetch a URL through Kadoa. Defaults to smart routing (auto fetch mode and proxy), can return page content as markdown for LLM consumption, and offers a stealth proxy tier for sites that block bots.

## Request body

- object
  - `url` string, uri, required
  - `method` 'GET' | 'POST' | 'PUT' | 'DELETE' | 'PATCH': HTTP method. Default: GET
  - `headers` object[]: Custom HTTP headers to send with the request
    - `key` string, required: Header name
    - `value` string, required: Header value
  - `body` union: Request body as JSON object, or a legacy worker-specific string. Prefer rawBody when the upstream endpoint needs an exact text payload.
    - object
    - string
  - `rawBody` string: Exact request body text to forward to the upstream URL. Use for endpoints that reject wrapper objects or require a specific JSON/form payload string.
  - `location` object: Geo-targeting for proxy selection
    - `country` string: ISO 3166-1 alpha-2 country code (e.g., 'US', 'GB', 'DE')
  - `timeout` integer: Per-attempt timeout in milliseconds
  - `overallTimeout` integer: Wall-clock deadline for the complete scrape in milliseconds. Default and maximum: 240000
  - `context` object: Sticky proxy session
    - `id` string: Sticky proxy session ID. Requests sharing an ID reuse the same proxy exit where the tier supports it.
  - `captureResponses` string[]: Substrings matched against network response URLs (browser workers only). The worker hooks every response the page makes and returns the body of each whose URL contains any of these substrings, in `capturedResponses`. Use this to read data that arrives via XHR/fetch and is NOT in the DOM — SPA dashboards (Tableau VizQL, Power BI) where values come back in a JSON/command response. Triggers a browser worker; pair with `browserActions` (clicks/waits) to drive the requests you want captured.
  - `waitUntil` 'load' | 'domcontentloaded' | 'networkidle' | 'commit': Browser-worker navigation gate. Default (omitted) is `domcontentloaded` + a bounded best-effort networkidle settle, which returns content even on heavy pages that never go fully idle (ASP.NET, SignalR, ad/telemetry-heavy). Set `networkidle` to block the whole navigation on full idle (old behavior — only when the page reliably goes idle and you need every lazy request). No-op for non-browser workers.
  - `settleMs` integer: Browser-worker cap (ms) for the best-effort networkidle settle after `domcontentloaded`. Default 15000. Raise for slow client-rendered pages that need more time for lazy content; lower to return faster. Ignored when `waitUntil: networkidle` and for non-browser workers.
  - `device` string: Device to emulate, by descriptor name (e.g. 'iPad', 'iPhone 13'). Honored by workers that support device emulation; ignored by the rest. Some sites only serve content for a given device profile.
  - `fetchMode` 'http' | 'browser' | 'auto': Fetch mode: http for fast requests, browser for JavaScript rendering and browser actions, or auto for smart routing that starts cheap and escalates when blocked. Default: auto
  - `proxy` 'auto' | 'direct' | 'dc' | 'isp' | 'residential' | 'stealth': Proxy tier: auto (smart selection), direct (Squid server egress without location guarantees), dc (datacenter), isp, residential, or stealth to route through the anti-bot providers for sites that block bots. Stealth providers use their own egress and pricing. Default: auto
  - `format` 'html' | 'markdown': Content format for HTML pages. markdown converts the page to markdown with navigation, header, and footer stripped — the compact shape to feed an LLM. Non-HTML responses (JSON, binary) and HTML pages over 2 MB are returned unchanged; check the response contentType to see which format was applied. Default: html
  - `browserActions` union[]
    - union: Browser action to perform on the page
      - object: Click an element on the page
        - `type` 'click', required: Action type
        - `selector` object, required: Element to click
          - `type` 'xpath' | 'css' | 'text', required: Selector type
          - `value` string, required: Selector value (e.g., '#submit-btn' for CSS, '//button[@type="submit"]' for XPath)
      - object: Type text into an input element
        - `type` 'input', required: Action type
        - `selector` object, required: Input element to type into
          - `type` 'xpath' | 'css' | 'text', required: Selector type
          - `value` string, required: Selector value (e.g., '#submit-btn' for CSS, '//button[@type="submit"]' for XPath)
        - `value` string, required: Text to type into the element
      - object: Scroll by a specific number of pixels
        - `type` 'scroll', required: Action type
        - `x` number: Horizontal scroll amount in pixels
        - `y` number, required: Vertical scroll amount in pixels
      - object: Scroll to the bottom of the page (useful for infinite scroll)
        - `type` 'scroll_to_bottom', required: Action type
        - `timeout_s` number: Maximum time to scroll in seconds (default: 30, maximum: 300)
      - object: Wait for a specified duration
        - `type` 'wait', required: Action type
        - `wait_time_s` number, required: Duration to wait in seconds (maximum: 300)
      - object: Capture a screenshot of the page
        - `type` 'screenshot', required: Action type
        - `full_page` boolean: Capture the full page (true) or just the viewport (false)
        - `format` 'jpg' | 'png' | 'webp': Output image format. Default: jpg
        - `quality` integer: Compression quality (1-100). Only applies to jpg and webp. Ignored for png. Default: 20
        - `width` integer: Resize to this width in pixels (height scales proportionally). No upscaling.
      - object: Execute JavaScript in the browser context
        - `type` 'evaluate_js', required: Action type
        - `script` string, required: JavaScript code to execute in the page context
      - object: Wait for an element to appear on the page
        - `type` 'wait_for_element', required: Action type
        - `selector` object, required: Element to wait for
          - `type` 'xpath' | 'css' | 'text', required: Selector type
          - `value` string, required: Selector value (e.g., '#submit-btn' for CSS, '//button[@type="submit"]' for XPath)
        - `timeout_s` number: Maximum time to wait in seconds (default: 10, maximum: 300)

## Response `200`

Scrape response with content and execution metadata

- object: Scrape response with content and execution metadata
  - `session` object: Acknowledged browser-session handle; pass it on subsequent raw API calls.
    - `id` string, required
  - `retryable` boolean: Whether repeating this scrape is safe and useful; absence does not authorize retry
  - `retryAfterMs` integer: Minimum delay before another scrape attempt, in milliseconds
  - `success` boolean, required: Whether the scrape succeeded
  - `contentType` string: MIME type of the response (e.g., text/html, application/json, image/png)
  - `content` string: Response content (raw text for html/json, base64 for binary)
  - `isBase64` boolean: Whether content is base64-encoded (true for binary content types like PDFs, images)
  - `screenshots` string[]: URLs of captured screenshots
  - `jsResults` unknown[]: Results of evaluate_js actions, in order of execution. Each entry is the JSON-serializable return value, or {error: string} if the script threw.
    - unknown
  - `capturedResponses` object[]: Bodies of network responses whose URL matched a `captureResponses` substring, in arrival order. Present only when `captureResponses` was set and at least one response matched.
    - `url` string, required: Full URL of the captured response
    - `status` number, required: HTTP status code of the captured response
    - `body` string, required: Response body as text
  - `statusCode` number: HTTP status code from the response
  - `headers` object: Response headers as key-value pairs
  - `cookies` object[]: Cookies set by the response
    - `name` string, required: Cookie name
    - `value` string, required: Cookie value
    - `domain` string: Cookie domain
    - `path` string: Cookie path
    - `expires` number: Expiration timestamp (Unix epoch)
    - `httpOnly` boolean: HttpOnly flag
    - `secure` boolean: Secure flag
    - `sameSite` 'Strict' | 'Lax' | 'None': SameSite attribute
  - `finalUrl` string: Final URL after any redirects
  - `execution` object, required: Execution metadata including timing and retry information
    - `requestId` string, uuid: Opaque request identifier for correlating scraper API and worker lifecycle diagnostics
    - `traceId` string: Opaque trace identifier for retrieving sanitized scrape timing diagnostics when tracing is enabled
    - `totalDurationMs` number, required: Total time including all retries in milliseconds
    - `contextId` string, nullable: Context ID used or created, if context feature was used
    - `attempts` object[], required
      - `worker` 'http' | 'browser' | 'stealth-http' | 'stealth-browser', required: Worker capability used for this attempt
      - `proxy` 'direct' | 'dc' | 'isp' | 'residential', nullable, required: Proxy type: direct Squid egress, datacenter, ISP, or residential
      - `location` object, nullable, required: Location used for geo-targeting, or null
        - `country` string: ISO 3166-1 alpha-2 country code (e.g., 'US', 'GB', 'DE')
      - `startedAt` string, date-time, required: ISO 8601 timestamp when attempt started
      - `endedAt` string, date-time, required: ISO 8601 timestamp when attempt ended
      - `durationMs` number, required: Attempt duration in milliseconds
      - `success` boolean, required: Whether this attempt succeeded
      - `provider` 'auto' | 'curl-cffi' | 'kurl' | 'kadmium' | 'scrape-ops' | 'driver': Concrete provider used for this attempt
      - `error` object: Error details if attempt failed
        - `code` 'BOT_DETECTED' | 'ACCESS_DENIED' | 'TIMEOUT' | 'JS_REQUIRED' | 'KYC_BLOCKED' | 'MAX_RETRIES_EXCEEDED' | 'NETWORK_ERROR' | 'SCREENSHOT_TOO_LARGE' | 'DEADLINE_EXCEEDED', required: Error code
        - `message` string, required: Human-readable error message
        - `appCode` string: Underlying backend application error code, when available
        - `provider` 'auto' | 'curl-cffi' | 'kurl' | 'kadmium' | 'scrape-ops' | 'driver': Concrete provider that produced the failure
        - `cluster` string: Cluster where the failure originated
        - `cause` object: Original failure when retries are exhausted
          - `code` 'BOT_DETECTED' | 'ACCESS_DENIED' | 'TIMEOUT' | 'JS_REQUIRED' | 'KYC_BLOCKED' | 'MAX_RETRIES_EXCEEDED' | 'NETWORK_ERROR' | 'SCREENSHOT_TOO_LARGE' | 'DEADLINE_EXCEEDED', required: Error code
          - `message` string, required: Human-readable error message
          - `appCode` string: Underlying backend application error code, when available
          - `provider` 'auto' | 'curl-cffi' | 'kurl' | 'kadmium' | 'scrape-ops' | 'driver': Concrete provider that produced the failure
          - `cluster` string: Cluster where the failure originated
  - `error` object: Error details when success is false
    - `code` 'BOT_DETECTED' | 'ACCESS_DENIED' | 'TIMEOUT' | 'JS_REQUIRED' | 'KYC_BLOCKED' | 'MAX_RETRIES_EXCEEDED' | 'NETWORK_ERROR' | 'SCREENSHOT_TOO_LARGE' | 'DEADLINE_EXCEEDED', required: Error code
    - `message` string, required: Human-readable error message
    - `appCode` string: Underlying backend application error code, when available
    - `provider` 'auto' | 'curl-cffi' | 'kurl' | 'kadmium' | 'scrape-ops' | 'driver': Concrete provider that produced the failure
    - `cluster` string: Cluster where the failure originated
    - `cause` object: Original failure when retries are exhausted
      - `code` 'BOT_DETECTED' | 'ACCESS_DENIED' | 'TIMEOUT' | 'JS_REQUIRED' | 'KYC_BLOCKED' | 'MAX_RETRIES_EXCEEDED' | 'NETWORK_ERROR' | 'SCREENSHOT_TOO_LARGE' | 'DEADLINE_EXCEEDED', required: Error code
      - `message` string, required: Human-readable error message
      - `appCode` string: Underlying backend application error code, when available
      - `provider` 'auto' | 'curl-cffi' | 'kurl' | 'kadmium' | 'scrape-ops' | 'driver': Concrete provider that produced the failure
      - `cluster` string: Cluster where the failure originated

## Other responses

- `400`: 400
- `401`: 401
- `403`: 403
- `429`: 429
- `500`: Internal server error
- `502`: 502
- `503`: The scrape backend or managed browser capacity is unavailable
- `504`: 504

---

[API](https://skmtc.dev/kadoa/apis/kadoa-api.md) · [All operations](https://skmtc.dev/kadoa/apis/kadoa-api/llms.txt) · [OpenAPI document](https://skmtc.dev/kadoa/apis/kadoa-api/revisions/bf7803c927b4?raw)