COcontext
Web Scraping

Scrape HTML

Changed on

Scrapes the given URL and returns the raw HTML content of the page. The base request costs 1 credit; requests with browser actions cost 2 credits.

get/web/scrape/html

Request

  • Base URL
    https://api.context.dev/v1
  • URL
    https://api.context.dev/v1/web/scrape/html
  • Auth
    HTTP bearer

Query parameters

urlstring uri required

Full URL to scrape (must include http:// or https:// protocol)

shouldParseboolean

When true, PDF URLs are fetched and parsed. When false, PDF URLs are skipped and a 400 PDF_SKIPPED is returned.

startinteger

First 1-based PDF page to parse. When omitted, parsing starts at the first page.

endinteger

Last 1-based PDF page to parse. When omitted, parsing ends at the last page. Must be greater than or equal to start when both are provided.

ocrboolean

When true, OCR the selected PDF pages that have no usable text layer (scans), replacing each recovered page's text with the OCR result while pages with a real text layer keep it. Billed at 1 credit per page OCR actually recovered, on top of the base request cost. When false, no OCR runs.

includeFramesboolean

When true, iframes are rendered inline into the returned HTML.

useMainContentOnlyboolean

When true, return only the page's main content in the HTML response, excluding headers, footers, sidebars, and navigation when detectable.

includeSelectorsstring[] nullable

CSS selectors. When provided, only matching subtrees (and their descendants) are kept and everything else is dropped. When omitted, the entire document is kept. Examples: "article.main", "#content", "[role=main]".

excludeSelectorsstring[] nullable

CSS selectors to remove from the result. Applied after includeSelectors. Exclusion takes precedence: an element matching both is removed. Examples: "nav", "footer", ".ad-banner", "[aria-hidden=true]".

maxAgeMsinteger nullable

Return a cached result if a prior scrape for the same parameters exists and is younger than this many milliseconds. Defaults to 1 day (86400000 ms) when omitted. Max is 30 days (2592000000 ms). Set to 0 to always scrape fresh.

waitForMsinteger nullable

Optional browser wait time in milliseconds after initial page load. Min: 0. Max: 30000 (30 seconds).

settleAnimationsboolean

When true, waits briefly for CSS and transition animations to settle before extracting HTML. Defaults to false. This adds a bit of latency in exchange for more stable output on animated pages.

headersobject

Optional outbound HTTP headers forwarded only to the target URL, sent as deep-object query params such as headers[X-Custom]=value. When provided, caching is bypassed: the result is neither read from nor written to cache.

country'ad' | 'ae' | 'af' | 'ag' | 'ai' | 'al' | 'am' | 'ao' | 'ar' | 'at' | 'au' | 'aw' | 'az' | 'ba' | 'bb' | 'bd' | 'be' | 'bf' | 'bg' | 'bh' | 'bi' | 'bj' | 'bm' | 'bn' | 'bo' | 'bq' | 'br' | 'bs' | 'bw' | 'by' | 'bz' | 'ca' | 'cd' | 'cf' | 'cg' | 'ch' | 'ci' | 'cl' | 'cm' | 'cn' | 'co' | 'cr' | 'cv' | 'cw' | 'cy' | 'cz' | 'de' | 'dj' | 'dk' | 'dm' | 'do' | 'dz' | 'ec' | 'ee' | 'eg' | 'es' | 'et' | 'fi' | 'fj' | 'fr' | 'ga' | 'gb' | 'gd' | 'ge' | 'gf' | 'gg' | 'gh' | 'gm' | 'gn' | 'gp' | 'gq' | 'gr' | 'gt' | 'gu' | 'gw' | 'gy' | 'hk' | 'hn' | 'hr' | 'ht' | 'hu' | 'id' | 'ie' | 'il' | 'im' | 'in' | 'iq' | 'ir' | 'is' | 'it' | 'je' | 'jm' | 'jo' | 'jp' | 'ke' | 'kg' | 'kh' | 'kn' | 'kr' | 'kw' | 'ky' | 'kz' | 'la' | 'lb' | 'lc' | 'lk' | 'lr' | 'ls' | 'lt' | 'lu' | 'lv' | 'ly' | 'ma' | 'mc' | 'md' | 'me' | 'mf' | 'mg' | 'mk' | 'ml' | 'mm' | 'mn' | 'mo' | 'mq' | 'mr' | 'mt' | 'mu' | 'mv' | 'mw' | 'mx' | 'my' | 'mz' | 'na' | 'nc' | 'ne' | 'ng' | 'ni' | 'nl' | 'no' | 'np' | 'nz' | 'om' | 'pa' | 'pe' | 'pf' | 'pg' | 'ph' | 'pk' | 'pl' | 'pr' | 'ps' | 'pt' | 'py' | 'qa' | 're' | 'ro' | 'rs' | 'ru' | 'rw' | 'sa' | 'sc' | 'sd' | 'se' | 'sg' | 'si' | 'sk' | 'sl' | 'sm' | 'sn' | 'so' | 'sr' | 'ss' | 'st' | 'sv' | 'sx' | 'sy' | 'sz' | 'tc' | 'td' | 'tg' | 'th' | 'tj' | 'tl' | 'tm' | 'tn' | 'tr' | 'tt' | 'tw' | 'tz' | 'ua' | 'ug' | 'us' | 'uy' | 'uz' | 'vc' | 've' | 'vg' | 'vi' | 'vn' | 'ye' | 'yt' | 'za' | 'zm' | 'zw'

Fetch the target page through a residential proxy in this country (ISO 3166-1 alpha-2).

Example:de
timeoutMSinteger

Optional timeout in milliseconds for the request. If the request takes longer than this value, it will be aborted with a 408 status code. Maximum allowed value is 300000ms (5 minutes).

zdr'enabled' | 'disabled'

Set to enabled to bypass shared caches and omit request and response content from retained usage logs. Requires zero data retention to be enabled for your organization (contact support@context.dev), otherwise the request fails with ZDR_NOT_ENABLED. Successful ZDR responses include X-Context-ZDR: true.

tagsstring[]

Optional tags for tracking usage. Up to 20 tags, each 1 to 50 characters.

Optional comma-separated caller-defined tags for tracking this request. Tags are recorded on the request's usage log and can be used to filter usage on the dashboard usage page. Up to 20 tags, each 1-50 characters.

[
  "production",
  "team-alpha"
]

Response

Successful response

successtrue required

Indicates success

htmlstring required

The scraped content of the page. For normal pages this is the raw HTML. When the page is a sitemap or feed served behind an XSL stylesheet (which browsers render into HTML), this is the underlying XML instead — see the type field.

urlstring required

The URL that was scraped

type'html' | 'xml' | 'json' | 'text' | 'csv' | 'markdown' | 'svg' | 'pdf' | 'docx' | 'doc' | 'xlsx' | 'xls' | 'pptx' | 'ppt' required

Detected content type of the returned html field. Sitemaps and feeds are surfaced as xml; ordinary pages are html. Excel workbooks are surfaced as xlsx/xls with the extracted sheets as HTML tables; PowerPoint presentations are surfaced as pptx/ppt with the extracted slides as HTML.

actionsHtmlStaleboolean

True when an action was applied but the returned content could not be refreshed afterward.

Changes

  • 1febe5476266Revision changes
    • info

      added the required property to the response with the status

  • 9eddb9e1a8eaRevision changes
    • breaking

      query request parameter includeFrames list-of-types was narrowed by removing types string

    • breaking

      query request parameter settleAnimations list-of-types was narrowed by removing types string

    • breaking

      query request parameter useMainContentOnly list-of-types was narrowed by removing types string

    • breaking

      property ocr of query request parameter pdf list-of-types was narrowed by removing types string

    • breaking

      property shouldParse of query request parameter pdf list-of-types was narrowed by removing types string

    • breaking

      for the query request parameter pdf, the type of property ocr was specialized from no type to boolean

    • breaking

      for the query request parameter pdf, the type of property shouldParse was specialized from no type to boolean

    • breaking

      for the query request parameter includeFrames, the type changed from no type to boolean

    • breaking

      for the query request parameter settleAnimations, the type changed from no type to boolean

    • breaking

      for the query request parameter useMainContentOnly, the type changed from no type to boolean