---
title: "Crawl Sitemap"
method: GET
path: "/web/scrape/sitemap"
tags: ["Web Scraping"]
---

# Crawl Sitemap

`GET /web/scrape/sitemap`

Crawl an entire website's sitemap and return all discovered page URLs.

## Query parameters

- `domain` string, required
- `maxLinks` integer
- `urlRegex` string
- `headers` HeaderMap
- `timeoutMS` integer — Optional timeout in milliseconds for the request. If the request takes longer than this value, it will be aborted with a 408 status code. Maximum allowed value is 300000ms (5 minutes).

## Response `200`

Successful response

- object
  - `success` true, required — Indicates success
  - `domain` string, required — The normalized domain that was crawled
  - `urls` string[], required — Array of discovered page URLs from the sitemap (max 500)
  - `meta` object, required — Metadata about the sitemap crawl operation
    - `sitemapsDiscovered` integer, required — Total number of sitemap files discovered
    - `sitemapsFetched` integer, required — Number of sitemap files successfully fetched and parsed
    - `sitemapsSkipped` integer, required — Number of sitemap files skipped (due to errors, timeouts, or limits)
    - `errors` integer, required — Number of errors encountered during crawling
  - `key_metadata` KeyMetadata — Metadata about the API key used for the request. Included in every response whenever a valid API key is provided, even when the response status is not 200.
    - `credits_consumed` integer, required — The number of credits consumed by this request.
    - `credits_remaining` integer, required — The number of credits remaining for your organization after this request.

## Other responses

- `400` — Bad request - Invalid domain
- `401` — Unauthorized - Invalid or missing API key
- `403` — Forbidden - Insufficient permissions or usage limit exceeded
- `408` — Request timeout
- `429` — Rate limit exceeded
- `500` — Internal server error

---

[API](https://skmtc.dev/context/apis/context-dev.md) · [All operations](https://skmtc.dev/context/apis/context-dev/llms.txt) · [OpenAPI document](https://skmtc-service-production.skmtc.workers.dev/v1/apis/context/context-dev/revisions/de91b92d5fb7/schema)