--- title: "Scrape URL" method: POST path: "/api/v1/scrape" tags: ["Web Scraping"] --- # Scrape URL `POST /api/v1/scrape` ## Request - Base URL: `https://api.finedata.ai` (the document declares no server; this is the origin it was published from) - URL: `https://api.finedata.ai/api/v1/scrape` - Auth: HTTP bearer Scrape a web page and return its HTML content. Send a URL and get back the rendered HTML, response headers, and metadata. Supports antibot bypass, JavaScript rendering, CAPTCHA solving, and residential proxies. **Minimal request:** `{"url": "https://example.com"}` ## Headers - `authorization` string, nullable - `x-api-key` string, nullable ## Request body - PublicScrapeRequest: Scrape request parameters. - `url` string, required: Target URL to scrape - `method` string: HTTP method (GET, POST, PUT, DELETE) - `headers` object, nullable: Custom HTTP headers to send with the request - `body` string, nullable: Request body (for POST/PUT requests) - `timeout` integer, nullable: Request timeout in seconds. Increase for captcha solving (max 300) - `max_retries` integer, nullable: Max automatic retry attempts on failure - `auto_retry` boolean: Automatically retry on failure (+1 token per retry) - `use_antibot` boolean: Browser-like TLS fingerprint (+2 tokens). Enabled by default - `tls_profile` string, nullable: TLS fingerprint profile: chrome136, chrome131, chrome124, firefox133, safari184, safari18_0, or 'vip' for premium rotation (vip:ios, vip:android, vip:windows, vip:mobile) - `use_residential` boolean: Route through residential proxy instead of datacenter (+3 tokens) - `use_isp` boolean: Route through ISP proxy for clean IP reputation (+2 tokens) - `use_mobile` boolean: Route through mobile carrier proxy for maximum anonymity (+4 tokens) - `proxy_sticky` boolean: Keep the same proxy exit IP for the entire browser session. Required when a warmup request and the follow-up fetch must come from one IP, e.g. a site that ties a session cookie to the address that obtained it. Works with residential proxies and BYOP profiles (rotation_mode=sticky). - `proxy_country` string, nullable: Request a proxy exit IP from a specific country (ISO-2 code, e.g. 'US', 'DE'). Applies to datacenter, residential, ISP and mobile proxies. Returns 422 if the country isn't currently supported for the resolved proxy type. Mutually exclusive with proxy_profile_id (BYOP). - `proxy_profile_id` integer, nullable: Use your own proxy list via a saved BYOP profile id (Bring Your Own Proxy). Mutually exclusive with use_residential / use_isp / use_mobile. Charges flat +1 token instead of the vendor surcharge. Requires team_s plan or admin override. - `stealth_antibot` boolean: Stronger browser emulation for pages that a plain request cannot load (+7 tokens) - `stealth_antibot_headful` boolean: Full real browser with a visible display, for pages the headless engine cannot load (+25 tokens) - `stealth_new` boolean: Experimental next-generation stealth engine (+15 tokens) - `stealth_premium` boolean: Premium stealth engine with full fingerprint emulation (+20 tokens) - `stealth_premium_headful` boolean: Premium stealth engine in headful mode with GPU rendering (+30 tokens) - `use_js_render` boolean: Render JavaScript with headless browser. Required for SPAs (+5 tokens) - `js_wait_for` string: Page load strategy: 'networkidle', 'load', 'domcontentloaded', or 'selector:.css-selector' - `js_scroll` boolean: Scroll to bottom to trigger lazy-loaded content - `js_actions` JsAction[], nullable: Sequence of browser actions before extracting HTML: click, wait, type, scroll, hover, evaluate (+2 tokens each, max 10) - `action` 'click' | 'wait' | 'wait_for' | 'scroll' | 'type' | 'hover' | 'screenshot' | 'evaluate', required: Action type: click, wait, wait_for, scroll, type, hover, screenshot, evaluate - `selector` string, nullable: CSS selector for the target element - `ms` integer, nullable: Wait time in milliseconds (for 'wait' action). Accepts both 'ms' and 'value' keys - `text` string, nullable: Text to type (for 'type' action) - `timeout` integer, nullable: Timeout for wait_for in ms - `y` integer, nullable: Y position for scroll (if no selector) - `script` string, nullable: JavaScript code to execute on the page (for 'evaluate' action). Runs in the page context with a 15s timeout. Max 10,000 characters - `solve_captcha` boolean: Auto-detect and solve captchas: reCAPTCHA, hCaptcha, Turnstile (+10 tokens) - `session_id` string, nullable: Sticky session ID. All requests with the same ID share the same proxy IP. Max 64 chars - `session_ttl` integer, nullable: Session lifetime in seconds (default: 1800 = 30 min, max: 86400 = 24h) - `formats` string[], nullable: Output formats to include in response. Options: 'markdown', 'rawHtml', 'text', 'links', 'screenshot', 'csv', 'xlsx'. csv/xlsx export AI extraction results as tabular data (requires extract_prompt or extract_schema). If not specified, only 'body' (raw HTML) is returned for backward compatibility. Example: ['markdown', 'links'] - `only_main_content` boolean: Extract only the main content of the page (removes navigation, footer, sidebar, ads). Applies to 'markdown' and 'text' formats. Uses Mozilla Readability algorithm - `raw_output` boolean: Return clean content directly instead of JSON wrapper. Requires exactly one format in 'formats' (e.g. formats=['markdown']). Response Content-Type matches the format: text/markdown, text/plain, text/html, image/png, text/csv, application/xlsx. Metadata is returned in response headers: X-Tokens-Used, X-Request-Id, X-Status-Code. Without 'formats', returns raw HTML as text/html. - `extract_rules` object, nullable: CSS/XPath selectors for structured data extraction. Simple: {'title': 'h1', 'price': '.price'}. Advanced: {'links': {'selector': 'a.product', 'type': 'list', 'output': '@href'}}. Supported types: text (default), html, list. Output: text content or @attr - `extract_schema` object, nullable: JSON Schema for AI-powered structured data extraction. The AI model analyzes page content and returns data matching your schema. Example: {'type': 'object', 'properties': {'products': {'type': 'array', 'items': {'type': 'object', 'properties': {'name': {'type': 'string'}, 'price': {'type': 'number'}}}}}} (+5 tokens) - `extract_prompt` string, nullable: Natural language instruction for AI-powered data extraction. Alternative to extract_schema. Example: 'Extract all product names, prices, and ratings from this page' (+5 tokens) - `ai_content_mode` 'full' | 'main', nullable: Content mode for AI extraction (extract_schema/extract_prompt). 'full' — AI sees full page content including sidebars, structured data blocks (best for product pages, real estate listings, marketplaces). 'main' — AI sees only main content extracted via Readability algorithm (best for articles, blog posts — fewer tokens, less noise). Default: 'full' ## Response `200` JSON response (default) or raw content when raw_output=true - PublicScrapeResponse: Scrape result. - `success` boolean, required: Whether the request completed successfully - `status_code` integer, required: HTTP status code of the target page - `headers` object, required: Response headers from the target page - `body` string, required: Raw HTML or text content of the page (kept for backward compatibility) - `data` ScrapeData: Structured scrape output. Fields are populated based on the requested 'formats'. - `markdown` string, nullable: Page content converted to Markdown format. Clean, LLM-friendly output - `raw_html` string, nullable: Original HTML content of the page - `text` string, nullable: Plain text content extracted from the page - `links` string[], nullable: All links (URLs) found on the page - `screenshot` string, nullable: Base64-encoded PNG screenshot of the page (requires JS rendering) - `extract` object, nullable: Structured data extracted via extract_rules, extract_schema, or extract_prompt - `metadata` object, nullable: Page metadata: title, description, language, og:image, favicon, etc. - `content_type` string, nullable: Original Content-Type of the response (useful for binary data detection) - `csv` string, nullable: AI extraction results as CSV string (when 'csv' format requested) - `xlsx` string, nullable: AI extraction results as base64-encoded XLSX (when 'xlsx' format requested) - `meta` object: Request metadata: url, resolved_url, elapsed_ms, proxy_country - `tokens_used` integer: Total tokens consumed for this request - `captcha_detected` boolean: Whether a captcha was detected on the page - `captcha_type` string, nullable: Captcha type if detected: recaptcha_v2, recaptcha_v3, hcaptcha, turnstile - `captcha_solved` boolean: Whether the captcha was solved successfully ## Other responses - `422`: Validation Error --- [API](https://skmtc.dev/finedata/apis/finedata-api.md) · [All operations](https://skmtc.dev/finedata/apis/finedata-api/llms.txt) · [OpenAPI document](https://skmtc.dev/finedata/apis/finedata-api/revisions/d4321b98fd0e?raw)