---
title: "Scrape a URL into markdown, HTML, links or metadata"
method: POST
path: "/web/scrape"
tags: ["Web"]
---

# Scrape a URL into markdown, HTML, links or metadata

`POST /web/scrape`

## Request

- Base URL: `https://api.crawlora.net/api/v1`
- URL: `https://api.crawlora.net/api/v1/web/scrape`
- Auth: API key in header `x-api-key`

Fetches a single public URL and returns clean content in the requested formats (markdown, html, raw_html, links, metadata). The request body IS the ScrapeOption object itself — e.g. {"url": "https://example.com"} — do not wrap it in an extra key. With render=auto the request starts as a fast HTTP fetch and escalates to a real browser when the page is blocked or rendered with JavaScript. only_main_content (default true) strips navigation, headers, footers and other boilerplate before conversion. Only public pages are supported; respect each site's terms of use and robots directives. A handful of popular sites (Amazon, Reddit, Yelp, LinkedIn, and others) already have a dedicated, more reliable endpoint elsewhere in this API — a failed scrape against one of them names it. A Reddit URL that maps onto a dedicated Reddit endpoint (listings, search, comment threads, user and domain pages) is answered by that endpoint instead, and its response is returned as-is in data with an X-Routed-To header; send dedicated=false to always get the scraped page.

## Headers

- `x-api-key` string, required

## Request body

- WebScrapeOption
  - `backend` string
  - `dedicated` boolean
  - `formats` string[]
  - `max_age` integer
  - `only_main_content` boolean
  - `proxy` string
  - `render` 'auto' | 'http' | 'browser' | 'unblocker'
  - `store_in_cache` boolean
  - `url` string, required
  - `wait_for` integer

## Response `200`

OK

- WebScrapeResponseDoc
  - `code` integer
  - `data` WebScrapeResult
    - `html` string
    - `link_details` WebScrapeLink[]
      - `href` string
      - `rel` string
      - `text` string
    - `links` string[]
    - `markdown` string
    - `metadata` WebScrapeMetadata
      - `author` string
      - `canonical_url` string
      - `content_type` string
      - `description` string
      - `final_url` string
      - `image` string
      - `language` string
      - `modified_at` string
      - `published_at` string
      - `section` string
      - `site_name` string
      - `source_url` string
      - `status_code` integer
      - `title` string
    - `raw_html` string
    - `scrape` WebScrapeInfo
      - `backend` string: headless backend that rendered, when method=browser
      - `cache_state` string: "hit" | "miss" — whether this response came from the cache
      - `cached_at` string: RFC3339 time the served copy was fetched (cache hits only)
      - `escalated` boolean: true when the fetch escalated from HTTP to a headless browser
      - `method` string: "http" | "browser"
      - `render_failed` boolean: RenderFailed is true when a browser render was requested (render=browser or a pinned backend) but no backend produced a page, so the content is the HTTP-tier fallback. Such a page ran no JavaScript and is billed at the HTTP-tier price.
  - `msg` string

## Other responses

- `400`: Invalid input
- `402`: Out of credits, or the plan's daily credit limit is used up. The body reports which limit was hit, when it resets, and how to raise it; retry_after_seconds matches the Retry-After header. Returned the same way by every credit-billed endpoint.
- `429`: Request rate exceeded for the plan. The body reports the plan's per-minute rate and how long to wait; retry_after_seconds matches the Retry-After header. Returned the same way by every credit-billed endpoint.
- `500`: Internal server error
- `503`: All fetch tiers were blocked or failed for the target URL

---

[API](https://skmtc.dev/crawlora/apis/crawlora-api.md) · [All operations](https://skmtc.dev/crawlora/apis/crawlora-api/llms.txt) · [OpenAPI document](https://skmtc.dev/crawlora/apis/crawlora-api/revisions/028417aa35bb?raw)