Web

Scrape a URL into markdown, HTML, links or metadata

Fetches a single public URL and returns clean content in the requested formats (markdown, html, raw_html, links, metadata). The request body IS the ScrapeOption object itself — e.g. {"url": "https://example.com"} — do not wrap it in an extra key. With render=auto the request starts as a fast HTTP fetch and escalates to a real browser when the page is blocked or rendered with JavaScript. only_main_content (default true) strips navigation, headers, footers and other boilerplate before conversion. Only public pages are supported; respect each site's terms of use and robots directives. A handful of popular sites (Amazon, Reddit, Yelp, LinkedIn, and others) already have a dedicated, more reliable endpoint elsewhere in this API — a failed scrape against one of them names it. A Reddit URL that maps onto a dedicated Reddit endpoint (listings, search, comment threads, user and domain pages) is answered by that endpoint instead, and its response is returned as-is in data with an X-Routed-To header; send dedicated=false to always get the scraped page.

post/web/scrape

Request

  • Base URL: https://api.crawlora.net/api/v1
  • URL: https://api.crawlora.net/api/v1/web/scrape
  • Auth: API key in header x-api-key

Headers

x-api-keystring required

API key required

Request body

backendstring
dedicatedboolean
formatsstring[]
max_ageinteger
only_main_contentboolean
proxystring
render'auto' | 'http' | 'browser' | 'unblocker'
store_in_cacheboolean
urlstring required
wait_forinteger

Example request

{
  "formats": [
    "markdown"
  ],
  "max_age": 604800000,
  "render": "auto",
  "url": "https://example.com"
}

Response

OK

codeinteger
msgstring

Example response

{
  "code": 200,
  "msg": "OK"
}

Changes

No recorded changes to this operation across all 2 revisions of this API.