---
title: "Map a site"
method: POST
path: "/v1/web/map"
tags: ["Crawl & batch"]
---

# Map a site

`POST /v1/web/map`

## Request

- Base URL: `https://api.hydrafetch.com`
- URL: `https://api.hydrafetch.com/v1/web/map`
- Auth: API key in header `X-API-Key`

Quickly enumerate a site's URLs without scraping them. Returns synchronously and costs one credit.

## Request body

- MapRequestDto
  - `url` string, required: The site whose URLs you want to enumerate. Must be http(s).
  - `includeLinks` boolean: Also include same-site links found on the starting page, not just the site's page list.
  - `limit` number: Maximum number of URLs to return.
  - `search` string: Keep only discovered URLs containing this term.
  - `sitemap` 'skip' | 'include' | 'only': Whether to use the site's published page list: skip it, include it alongside discovered links, or use it only.
  - `sitemapInclude` string: On sitemap-index sites, only descend into sub-sitemaps whose URL contains one of these comma-separated terms (case-insensitive). E.g. "post" to target article sitemaps.
  - `sitemapExclude` string: Skip sub-sitemaps whose URL contains any of these comma-separated terms (case-insensitive). Compose with sitemapInclude — e.g. include "post" + exclude "tag,category,author" avoids WordPress taxonomy sitemaps (post_tag-sitemap, …-taxonomies-post_tag-*) that also contain "post".
  - `order` 'newest' | 'oldest' | 'sitemap': Sort order by sitemap lastmod. "newest" (default) returns the most recent URLs first — the wanted end of a large archive under the limit. URLs without a lastmod sort last.
  - `before` string: Cursor for paging a large archive newest→oldest: return only URLs with lastmod at or before this ISO timestamp. Page by passing the oldest lastmod from the previous page; dedupe by url across pages.
  - `offset` number: Skip the first N results (in the sorted order) then return the next `limit`. Pagination for sources with no lastmod (WordPress-core), where the date cursor can't apply. Page sequentially (0, limit, 2·limit, …) so the sub-sitemap cache keeps deep pages fast.
  - `includeSubdomains` boolean: Also include URLs on subdomains of the site. Default off.
  - `ignoreQueryParameters` boolean: Treat URLs that differ only by query string as one. Default off.

## Response `200`

The discovered URLs.

- MapResponseDto
  - `data` MapResultDto, required
    - `url` string, required: The site you requested.
    - `links` MapUrlEntryDto[], required: The site's discovered URLs, each with its sitemap lastmod (null when unknown).
      - `url` string, required
      - `lastmod` object, nullable, required: The <lastmod> from the sitemap, if present. null for links found on the page.
    - `count` number, required: How many URLs were returned.
    - `total` number, required: Total URLs in the enumeration this page is a slice of. Page with `offset` until offset+count reaches this — do not infer the end from a short page.
    - `truncated` boolean, required: Whether enumeration hit its ceiling or time budget. When true, `total` is a floor rather than the real size of the site.

## Other responses

- `400`: The request body or query failed validation.
- `401`: The X-API-Key header is missing, malformed, or names a key that no longer exists.
- `402`: The workspace has no credits left for this request. Nothing was charged and nothing was fetched.
- `403`: The key is valid but the workspace may not make this request.
- `404`: No such job, or the id belongs to another workspace.
- `422`: The target site did not respond, or we could not read that page. This is a fact about that URL rather than a fault on our side, and retrying it will not change the answer. When details.retryAfterSeconds is present, asking again before then returns this same answer. Nothing was charged.
- `429`: The workspace exceeded its per-minute request limit. Read the RateLimit headers, or Retry-After on this response, and retry after the window resets.
- `500`: Unexpected server error. Nothing was charged.
- `503`: A dependency needed for the requested formats is unavailable. Safe to retry.

---

[API](https://skmtc.dev/hydrafetch/apis/hydrafetch-api.md) · [All operations](https://skmtc.dev/hydrafetch/apis/hydrafetch-api/llms.txt) · [OpenAPI document](https://skmtc.dev/hydrafetch/apis/hydrafetch-api/revisions/b28322984b4b?raw)