--- title: "Crawl a website" method: POST path: "/web/crawl" tags: ["Web Scraping"] --- # Crawl a website `POST /web/crawl` ## Request - Base URL: `https://api.context.dev/v1` - URL: `https://api.context.dev/v1/web/crawl` - Auth: HTTP bearer Crawl a website and return page content as Markdown. Use a batch for crawls beyond 500 pages. ## Request body - object - `url` string, uri, required: Start URL, including `http://` or `https://`. - `maxPages` integer: Maximum pages to crawl. - `maxDepth` integer: Maximum link depth from the starting URL (0 = only the starting page) - `urlRegex` string: Regex pattern. Only URLs matching this pattern will be followed and scraped. An automatic prefix scope in the form ^ follows a redirect of the starting page. - `excludeUrlRegex` string: Regex pattern. URLs matching this pattern are never followed or scraped, even when they match urlRegex. The starting URL is always scraped. - `includeLinks` boolean: Preserve hyperlinks in the Markdown output (`html-conversion` only). - `includeImages` boolean: Include image references in the Markdown output (`html-conversion` only). - `shortenBase64Images` boolean: Truncate base64-encoded image data in the Markdown output (`html-conversion` only). - `useMainContentOnly` boolean: Extract only the main content, stripping headers, footers, sidebars, and navigation - `sourcePreference` string[]: Markdown sources in order of preference: `html-conversion` converts the page HTML, `markdown-direct` uses Markdown the site publishes. - `followSubdomains` boolean: When true, follow links on subdomains of the starting URL's domain (e.g. docs.example.com when starting from example.com). www and apex are always treated as equivalent. - `pdf` object: PDF handling. `start`/`end` limit parsing to an inclusive, 1-based page range. - `shouldParse` boolean: When true, PDF pages are fetched and parsed. When false, PDF pages are skipped entirely (not included in results and not counted as failures). - `start` integer: First 1-based PDF page to parse. - `end` integer: Last PDF page to parse (1-based, inclusive). Defaults to the final page. Must be >= start. - `ocr` boolean: Read scanned PDF pages with OCR; preserve pages that already have text. - `includeFrames` boolean: When true, the contents of iframes are rendered to Markdown for each crawled page. - `includeSelectors` string[], nullable: Keep matching HTML subtrees before converting each page to Markdown. - `excludeSelectors` string[], nullable: Remove matching elements after inclusions. Exclusions take precedence. - `maxAgeMs` integer, nullable: Maximum cache age in milliseconds. Defaults to 1 day; `0` fetches fresh. - `waitForMs` integer, nullable: Browser wait time in milliseconds after initial page load for each crawled page. Defaults to 3500 (3.5 seconds). Min: 0. Max: 30000 (30 seconds). - `settleAnimations` boolean: Wait briefly for CSS animations and transitions to settle before reading each page. - `stopAfterMs` integer: Soft crawl deadline in milliseconds. Returns pages collected before the next deadline check. - `timeoutOpts` PartialTimeout: Request deadline and what to return when it passes. - `milliseconds` integer, required: Deadline in milliseconds. - `behavior` 'fail' | 'return-partial': "fail" returns 408 at the deadline. "return-partial" returns available results; inspect the response’s partial flag. - `country` 'ad' | 'ae' | 'af' | 'ag' | 'ai' | 'al' | 'am' | 'ao' | 'ar' | 'at' | 'au' | 'aw' | 'az' | 'ba' | 'bb' | 'bd' | 'be' | 'bf' | 'bg' | 'bh' | 'bi' | 'bj' | 'bm' | 'bn' | 'bo' | 'bq' | 'br' | 'bs' | 'bw' | 'by' | 'bz' | 'ca' | 'cd' | 'cf' | 'cg' | 'ch' | 'ci' | 'cl' | 'cm' | 'cn' | 'co' | 'cr' | 'cv' | 'cw' | 'cy' | 'cz' | 'de' | 'dj' | 'dk' | 'dm' | 'do' | 'dz' | 'ec' | 'ee' | 'eg' | 'es' | 'et' | 'fi' | 'fj' | 'fr' | 'ga' | 'gb' | 'gd' | 'ge' | 'gf' | 'gg' | 'gh' | 'gm' | 'gn' | 'gp' | 'gq' | 'gr' | 'gt' | 'gu' | 'gw' | 'gy' | 'hk' | 'hn' | 'hr' | 'ht' | 'hu' | 'id' | 'ie' | 'il' | 'im' | 'in' | 'iq' | 'ir' | 'is' | 'it' | 'je' | 'jm' | 'jo' | 'jp' | 'ke' | 'kg' | 'kh' | 'kn' | 'kr' | 'kw' | 'ky' | 'kz' | 'la' | 'lb' | 'lc' | 'lk' | 'lr' | 'ls' | 'lt' | 'lu' | 'lv' | 'ly' | 'ma' | 'mc' | 'md' | 'me' | 'mf' | 'mg' | 'mk' | 'ml' | 'mm' | 'mn' | 'mo' | 'mq' | 'mr' | 'mt' | 'mu' | 'mv' | 'mw' | 'mx' | 'my' | 'mz' | 'na' | 'nc' | 'ne' | 'ng' | 'ni' | 'nl' | 'no' | 'np' | 'nz' | 'om' | 'pa' | 'pe' | 'pf' | 'pg' | 'ph' | 'pk' | 'pl' | 'pr' | 'ps' | 'pt' | 'py' | 'qa' | 're' | 'ro' | 'rs' | 'ru' | 'rw' | 'sa' | 'sc' | 'sd' | 'se' | 'sg' | 'si' | 'sk' | 'sl' | 'sm' | 'sn' | 'so' | 'sr' | 'ss' | 'st' | 'sv' | 'sx' | 'sy' | 'sz' | 'tc' | 'td' | 'tg' | 'th' | 'tj' | 'tl' | 'tm' | 'tn' | 'tr' | 'tt' | 'tw' | 'tz' | 'ua' | 'ug' | 'us' | 'uy' | 'uz' | 'vc' | 've' | 'vg' | 'vi' | 'vn' | 'ye' | 'yt' | 'za' | 'zm' | 'zw': Fetch from this country (ISO 3166-1 alpha-2). - `zdr` 'enabled' | 'disabled': `enabled` turns on zero data retention. Returns 403 `ZDR_NOT_ENABLED` unless your organization has ZDR. - `tags` string[]: Labels for filtering usage in the dashboard. ## Response `200` Successful response - object - `partial` boolean: True when timeoutOpts.behavior=return-partial returned the usable results collected before the deadline. Partial collections are not cached as complete results. - `results` object[], required - `markdown` string, required: Extracted page content as Markdown (empty string on failure) - `metadata` object, required: Metadata extracted from the scraped page HTML. - `sourceUrl` string, required: Original URL requested by the caller. - `finalUrl` string, required: Final URL scraped after redirects or scraper fallback, when known. Falls back to sourceUrl when unavailable. - `title` string, required: Best page title extracted from the page (empty string if unavailable). - `description` string: Best description extracted from standard, Open Graph, or Twitter metadata. - `language` string: Language extracted from html lang or language meta tags. - `keywords` string[]: Keywords extracted from the page's keywords meta tag. - `canonicalUrl` string: Resolved canonical URL, when present. - `author` string: Author metadata, when present. - `siteName` string: Site or application name from page metadata. - `image` string: Primary resolved preview image from Open Graph, Twitter, or image metadata. - `favicon` string: Resolved favicon URL, when present. - `publishedTime` string: Published timestamp/date from page metadata, when present. - `modifiedTime` string: Modified timestamp/date from page metadata, when present. - `robots` string: Robots meta directive, when present. - `openGraph` object: Open Graph metadata with the og: prefix removed and keys camel-cased. - `twitter` object: Twitter card metadata with the twitter: prefix removed and keys camel-cased. - `alternates` PageMetadataAlternate[]: Resolved alternate links from link rel=alternate tags. - `href` string, required: Resolved alternate URL. - `hreflang` string: Language or locale for the alternate URL, when present. - `type` string: Alternate resource MIME type, when present. - `title` string: Alternate resource title, when present. - `headings` PageMetadataHeading[]: Page headings (h1–h6) in document order, extracted from the unfiltered document. Capped at the first 500 headings. Omitted when the page has none. - `level` integer, required: Heading level, 1–6 (from h1–h6). - `text` string, required: Heading text with whitespace collapsed, truncated to 1000 characters. - `jsonLd` object[]: JSON-LD structured data blocks parsed from the page. - `additionalMeta` object: Additional non-social meta tags not promoted to top-level metadata fields. - `url` string, required: The crawl URL fetched for this page. - `crawlDepth` integer, required: Depth relative to the start URL. 0 = start URL, 1 = one link away. - `statusCode` integer, required: HTTP status code of the response - `success` boolean, required: true if the page was fetched and parsed successfully - `metadata` object, required - `numUrls` integer, required: Total number of URLs crawled - `maxCrawlDepth` integer, required: Maximum crawl depth reached during the crawl - `numSucceeded` integer, required: Number of pages successfully crawled - `numFailed` integer, required: Number of pages that failed to crawl - `numSkipped` integer, required: Number of URLs skipped (PDFs when pdf.shouldParse=false, URLs not matching urlRegex, or URLs matching excludeUrlRegex) - `cache_metadata` CacheMetadata, required: Whether this response came from cache. - `status` 'hit' | 'miss' | 'zdr', required: Whether the response was served from cache, required fresh work, or honored zero-data-retention cache bypass. - `age_ms` integer, required: Age of the cached data in milliseconds. Zero for miss and zdr responses. - `key_metadata` KeyMetadata: Credits this request used and your remaining balance. - `credits_consumed` integer, required: Credits charged for this request. - `credits_remaining` integer, required: Credits remaining for your organization. - `request_id` string, uuid, required: Unique ID of this request, also in `X-Request-Id`. Include it when contacting support. ## Other responses - `400`: Bad request - Invalid URL or parameters - `401`: Missing, invalid, or disabled API key. - `402`: Insufficient credits (`USAGE_EXCEEDED`); authentication is valid. Retry once funded. Legacy deployments may return 401. - `403`: API key permissions or organization settings do not allow this request. - `404`: Start URL returned a 404 - `408`: Request timeout - `415`: The start URL has an unsupported content type. Supported documents include PDF, Word, Excel, PowerPoint, and CSV. - `429`: Rate limit exceeded. Organization concurrency rejections include structured error_data and subscription settings; follow Retry-After before retrying. - `500`: Internal server error ## Changes - **2026-10-21** `61627728926b`: 3 warning, 1 info - added the new `NO_PUBLISHED_MARKDOWN` enum value to the `error_code` response property for the response status `401` - added the new `NO_PUBLISHED_MARKDOWN` enum value to the `error_code` response property for the response status `402` - added the new `NO_PUBLISHED_MARKDOWN` enum value to the `error_code` response property for the response status `403` - added the new optional request property `sourcePreference` - **2026-10-19** `aeb64a2b7925`: 1 info - added the new optional request property `excludeUrlRegex` - **2026-10-19** `4ddcf49b9fe5`: 4 info - added the non-success response with the status `402` - added the optional property `error_data` to the response with the `401` status - added the optional property `error_data` to the response with the `403` status - added the optional property `error_data` to the response with the `429` status - **2026-10-16** `2125f3e94e52`: 1 info - endpoint added [Operation history](https://skmtc.dev/context/apis/brand-api/changes/web/crawl/post.md) --- [API](https://skmtc.dev/context/apis/brand-api.md) · [All operations](https://skmtc.dev/context/apis/brand-api/llms.txt) · [OpenAPI document](https://skmtc.dev/context/apis/brand-api/revisions/9e2ab7841fb1?raw)