Web Scraping

Map URLs

Changed on

Discover a site's URLs, with page titles, descriptions, keywords, and language when available. Metadata can be missing on newly discovered URLs.

get/web/urls

Request

  • Base URL: https://api.context.dev/v1
  • URL: https://api.context.dev/v1/web/urls
  • Auth: HTTP bearer

Query parameters

domainstring required

Domain to map, e.g. stripe.com.

Example:stripe.com
includeSubdomainsboolean

Include URLs on subdomains.

maxLinksinteger

Maximum number of URLs to return.

sitemapUrlstring uri

Fetch this sitemap instead of discovering sitemaps. Must belong to the domain or a subdomain.

urlRegexstring

Regex pattern. Only URLs matching this pattern are returned and counted against maxLinks.

Example:^https?://[^/]+/blog/
excludeUrlRegexstring

Regex pattern. URLs matching this pattern are not returned, even when they match urlRegex.

Example:/tag/|/author/
searchstring

Filter URLs by a topic or phrase, most relevant first.

Example:help center and troubleshooting articles
headersobject

HTTP headers for the target origin. Non-empty headers bypass caching.

millisecondsinteger required

Deadline in milliseconds.

behavior'fail' | 'return-partial'

"fail" returns 408 at the deadline. "return-partial" returns available results; inspect the response’s partial flag.

zdr'enabled' | 'disabled'

enabled turns on zero data retention. Returns 403 ZDR_NOT_ENABLED unless your organization has ZDR.

tagsstring[]

Labels for filtering usage in the dashboard.

Comma-separated labels for filtering usage, e.g. production,team-alpha.

[
  "production",
  "team-alpha"
]

Response

Discovered URLs with available page metadata

successtrue required
domainstring required
partialboolean
request_idstring uuid required

Unique ID of this request, also in X-Request-Id. Include it when contacting support.

Example response

{
  "request_id": "3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91"
}

Changes

  • 61627728926bRevision changes
    • warning

      added the new NO_PUBLISHED_MARKDOWN enum value to the response property for the response status

    • warning

      added the new NO_PUBLISHED_MARKDOWN enum value to the response property for the response status

    • warning

      added the new NO_PUBLISHED_MARKDOWN enum value to the response property for the response status

    • warning

      added the new NO_PUBLISHED_MARKDOWN enum value to the response property for the response status

  • aeb64a2b7925Revision changes
    • info

      added the new optional query request parameter excludeUrlRegex