Web Scraping

Crawl a website

Changed on

Crawl a website and return page content as Markdown. Use a batch for crawls beyond 500 pages.

post/web/crawl

Request

  • Base URL: https://api.context.dev/v1
  • URL: https://api.context.dev/v1/web/crawl
  • Auth: HTTP bearer

Request body

urlstring uri required

Start URL, including http:// or https://.

maxPagesinteger

Maximum pages to crawl.

maxDepthinteger

Maximum link depth from the starting URL (0 = only the starting page)

urlRegexstring

Regex pattern. Only URLs matching this pattern will be followed and scraped. An automatic prefix scope in the form ^<starting URL> follows a redirect of the starting page.

excludeUrlRegexstring

Regex pattern. URLs matching this pattern are never followed or scraped, even when they match urlRegex. The starting URL is always scraped.

includeLinksboolean

Preserve hyperlinks in the Markdown output (html-conversion only).

includeImagesboolean

Include image references in the Markdown output (html-conversion only).

shortenBase64Imagesboolean

Truncate base64-encoded image data in the Markdown output (html-conversion only).

useMainContentOnlyboolean

Extract only the main content, stripping headers, footers, sidebars, and navigation

sourcePreferencestring[]

Markdown sources in order of preference: html-conversion converts the page HTML, markdown-direct uses Markdown the site publishes.

followSubdomainsboolean

When true, follow links on subdomains of the starting URL's domain (e.g. docs.example.com when starting from example.com). www and apex are always treated as equivalent.

includeFramesboolean

When true, the contents of iframes are rendered to Markdown for each crawled page.

includeSelectorsstring[] nullable

Keep matching HTML subtrees before converting each page to Markdown.

excludeSelectorsstring[] nullable

Remove matching elements after inclusions. Exclusions take precedence.

maxAgeMsinteger nullable

Maximum cache age in milliseconds. Defaults to 1 day; 0 fetches fresh.

waitForMsinteger nullable

Browser wait time in milliseconds after initial page load for each crawled page. Defaults to 3500 (3.5 seconds). Min: 0. Max: 30000 (30 seconds).

settleAnimationsboolean

Wait briefly for CSS animations and transitions to settle before reading each page.

stopAfterMsinteger

Soft crawl deadline in milliseconds. Returns pages collected before the next deadline check.

country'ad' | 'ae' | 'af' | 'ag' | 'ai' | 'al' | 'am' | 'ao' | 'ar' | 'at' | 'au' | 'aw' | 'az' | 'ba' | 'bb' | 'bd' | 'be' | 'bf' | 'bg' | 'bh' | 'bi' | 'bj' | 'bm' | 'bn' | 'bo' | 'bq' | 'br' | 'bs' | 'bw' | 'by' | 'bz' | 'ca' | 'cd' | 'cf' | 'cg' | 'ch' | 'ci' | 'cl' | 'cm' | 'cn' | 'co' | 'cr' | 'cv' | 'cw' | 'cy' | 'cz' | 'de' | 'dj' | 'dk' | 'dm' | 'do' | 'dz' | 'ec' | 'ee' | 'eg' | 'es' | 'et' | 'fi' | 'fj' | 'fr' | 'ga' | 'gb' | 'gd' | 'ge' | 'gf' | 'gg' | 'gh' | 'gm' | 'gn' | 'gp' | 'gq' | 'gr' | 'gt' | 'gu' | 'gw' | 'gy' | 'hk' | 'hn' | 'hr' | 'ht' | 'hu' | 'id' | 'ie' | 'il' | 'im' | 'in' | 'iq' | 'ir' | 'is' | 'it' | 'je' | 'jm' | 'jo' | 'jp' | 'ke' | 'kg' | 'kh' | 'kn' | 'kr' | 'kw' | 'ky' | 'kz' | 'la' | 'lb' | 'lc' | 'lk' | 'lr' | 'ls' | 'lt' | 'lu' | 'lv' | 'ly' | 'ma' | 'mc' | 'md' | 'me' | 'mf' | 'mg' | 'mk' | 'ml' | 'mm' | 'mn' | 'mo' | 'mq' | 'mr' | 'mt' | 'mu' | 'mv' | 'mw' | 'mx' | 'my' | 'mz' | 'na' | 'nc' | 'ne' | 'ng' | 'ni' | 'nl' | 'no' | 'np' | 'nz' | 'om' | 'pa' | 'pe' | 'pf' | 'pg' | 'ph' | 'pk' | 'pl' | 'pr' | 'ps' | 'pt' | 'py' | 'qa' | 're' | 'ro' | 'rs' | 'ru' | 'rw' | 'sa' | 'sc' | 'sd' | 'se' | 'sg' | 'si' | 'sk' | 'sl' | 'sm' | 'sn' | 'so' | 'sr' | 'ss' | 'st' | 'sv' | 'sx' | 'sy' | 'sz' | 'tc' | 'td' | 'tg' | 'th' | 'tj' | 'tl' | 'tm' | 'tn' | 'tr' | 'tt' | 'tw' | 'tz' | 'ua' | 'ug' | 'us' | 'uy' | 'uz' | 'vc' | 've' | 'vg' | 'vi' | 'vn' | 'ye' | 'yt' | 'za' | 'zm' | 'zw'

Fetch from this country (ISO 3166-1 alpha-2).

zdr'enabled' | 'disabled'

enabled turns on zero data retention. Returns 403 ZDR_NOT_ENABLED unless your organization has ZDR.

tagsstring[]

Labels for filtering usage in the dashboard.

Example request

{
  "urlRegex": "^https?://[^/]+/blog/",
  "excludeUrlRegex": "/tag/|/author/",
  "country": "de",
  "tags": [
    "production",
    "team-alpha"
  ]
}

Response

Successful response

partialboolean

True when timeoutOpts.behavior=return-partial returned the usable results collected before the deadline. Partial collections are not cached as complete results.

request_idstring uuid required

Unique ID of this request, also in X-Request-Id. Include it when contacting support.

Example response

{
  "request_id": "3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91"
}

Changes

  • 61627728926bRevision changes
    • warning

      added the new NO_PUBLISHED_MARKDOWN enum value to the response property for the response status

    • warning

      added the new NO_PUBLISHED_MARKDOWN enum value to the response property for the response status

    • warning

      added the new NO_PUBLISHED_MARKDOWN enum value to the response property for the response status

    • info

      added the new optional request property

  • aeb64a2b7925Revision changes
    • info

      added the new optional request property