COcontext
Web Scraping

Crawl Sitemap

Changed on

Crawl an entire website's sitemap and return all discovered page URLs. Pass search to have the crawled sitemap filtered down to the pages about a phrase (for example pricing and plans or api authentication docs), most relevant first — a searched crawl scans the whole sitemap and costs 2 credits instead of 1.

get/web/scrape/sitemap

Request

  • Base URL
    https://api.context.dev/v1
  • URL
    https://api.context.dev/v1/web/scrape/sitemap
  • Auth
    HTTP bearer

Query parameters

domainstring required

Domain to build a sitemap for

maxLinksinteger

Maximum number of links to return from the sitemap crawl. Defaults to 10,000. Minimum is 1, maximum is 100,000.

sitemapUrlstring uri

Optional explicit sitemap URL. When provided, exactly this sitemap is crawled instead of discovering the domain's sitemaps.

urlRegexstring

Optional RE2-compatible regex pattern. Only URLs matching this pattern are returned and counted against maxLinks.

Example:^https?://[^/]+/blog/
searchstring

Optional search phrase. When provided, the crawled sitemap is filtered to the pages whose URLs are about that phrase, most relevant first, and the request costs 2 credits instead of 1.

Example:help center and troubleshooting articles
headersobject

Optional outbound HTTP headers forwarded only to the target URL, sent as deep-object query params such as headers[X-Custom]=value. When provided, caching is bypassed: the result is neither read from nor written to cache.

timeoutMSinteger

Optional timeout in milliseconds for the request. If the request takes longer than this value, it will be aborted with a 408 status code. Maximum allowed value is 300000ms (5 minutes).

zdr'enabled' | 'disabled'

Set to enabled to bypass shared caches and omit request and response content from retained usage logs. Requires zero data retention to be enabled for your organization (contact support@context.dev), otherwise the request fails with ZDR_NOT_ENABLED. Successful ZDR responses include X-Context-ZDR: true.

tagsstring[]

Optional tags for tracking usage. Up to 20 tags, each 1 to 50 characters.

Optional comma-separated caller-defined tags for tracking this request. Tags are recorded on the request's usage log and can be used to filter usage on the dashboard usage page. Up to 20 tags, each 1-50 characters.

[
  "production",
  "team-alpha"
]

Response

Successful response

successtrue required

Indicates success

domainstring required

The normalized domain that was crawled

urlsstring[] required

Discovered page URLs from the sitemap, up to maxLinks. When search is set these are only the matching pages, most relevant first.

Changes

  • fc8ac2f5f021Revision changes
    • info

      added the new optional query request parameter search

  • 3764d9c86493Revision changes
    • breaking

      for the query request parameter domain, the minLength was increased from 0 to 3

    • info

      added the new optional query request parameter sitemapUrl

    • info

      added the new optional query request parameter tags

    • info

      added the new optional query request parameter zdr

    • info

      for the query request parameter timeoutMS, the min was decreased from 1000.00 to 1.00