---
title: "Crawl website sitemap"
method: GET
path: "/web/scrape/sitemap"
tags: ["Web Scraping"]
---

# Crawl website sitemap

`GET /web/scrape/sitemap`

Crawls the sitemap of the given domain and returns all discovered page URLs. Supports sitemap index files (recursive), parallel fetching with concurrency control, deduplication, and filters out non-page resources (images, PDFs, etc.).

## Query parameters

- `domain` string, required
- `maxLinks` integer

## Response `200`

Successful response

- object
  - `success` true, required — Indicates success
  - `domain` string, required — The normalized domain that was crawled
  - `urls` string[], required — Array of discovered page URLs from the sitemap (max 500)
  - `meta` object, required — Metadata about the sitemap crawl operation
    - `sitemapsDiscovered` integer, required — Total number of sitemap files discovered
    - `sitemapsFetched` integer, required — Number of sitemap files successfully fetched and parsed
    - `sitemapsSkipped` integer, required — Number of sitemap files skipped (due to errors, timeouts, or limits)
    - `errors` integer, required — Number of errors encountered during crawling

## Other responses

- `400` — Bad request - Invalid domain
- `401` — Unauthorized - Invalid or missing API key
- `403` — Forbidden - Insufficient permissions or usage limit exceeded
- `408` — Request timeout
- `500` — Internal server error

## Changes

- **2026-03-25** `1cd00999a087` — 1 info
  - added the new optional `query` request parameter `maxLinks`
- **2026-02-23** `2cdd67823c6a` — 1 info
  - endpoint added

[Change history](https://skmtc.dev/context/apis/brand-api/changes/web/scrape/sitemap/get.md)

---

[API](https://skmtc.dev/context/apis/brand-api.md) · [All operations](https://skmtc.dev/context/apis/brand-api/llms.txt) · [OpenAPI document](https://skmtc-service-production.skmtc.workers.dev/v1/apis/context/brand-api/revisions/05a30711e18b/schema)