Web: Scrape

Scrape a web page

Fetches a single URL and returns its content. Default is clean main-content markdown — cheapest and right for almost everything; just read the markdown. Reach for the pricier options only when the task truly needs them: the json format with a schema runs an LLM to extract structured fields (costs several credits per page — prefer scraping markdown and parsing it in code), and proxy: enhanced forces heavy anti-bot handling. Scrape only the specific pages you will actually use. Handles JavaScript-rendered pages.

post/v1/data/web/scrape

Request

  • Base URL: https://api.superagnt.com
  • URL: https://api.superagnt.com/v1/data/web/scrape
  • Auth: API key in header Authorization

Request body

urlstring uri required

The URL to scrape (must be a publicly reachable http(s) URL).

onlyMainContentboolean

Return only the main content of the page, excluding navs, headers, and footers.

includeTagsstring[]

Only include content within these HTML tags / CSS selectors.

excludeTagsstring[]

Exclude content within these HTML tags / CSS selectors.

maxAgeinteger

Return cached content up to this age in milliseconds (faster, cheaper). Default 172800000 (2 days).

waitForinteger

Milliseconds to wait for the page to render before scraping.

mobileboolean

Emulate a mobile device.

timeoutinteger

Request timeout in milliseconds (1000-300000). Default 60000.

proxy'basic' | 'enhanced' | 'auto'

Proxy / anti-bot mode. 'auto' (default) escalates only on failure; 'enhanced' forces advanced anti-bot handling and costs more.

Response

The scraped page content in the requested formats.

object required

Changes

No recorded changes to this operation across all 1 revision of this API.