Crawl & batch

Map a site

Quickly enumerate a site's URLs without scraping them. Returns synchronously and costs one credit.

post/v1/web/map

Request

  • Base URL: https://api.hydrafetch.com
  • URL: https://api.hydrafetch.com/v1/web/map
  • Auth: API key in header X-API-Key

Request body

urlstring required

The site whose URLs you want to enumerate. Must be http(s).

includeLinksboolean

Also include same-site links found on the starting page, not just the site's page list.

limitnumber

Maximum number of URLs to return.

searchstring

Keep only discovered URLs containing this term.

sitemap'skip' | 'include' | 'only'

Whether to use the site's published page list: skip it, include it alongside discovered links, or use it only.

sitemapIncludestring

On sitemap-index sites, only descend into sub-sitemaps whose URL contains one of these comma-separated terms (case-insensitive). E.g. "post" to target article sitemaps.

sitemapExcludestring

Skip sub-sitemaps whose URL contains any of these comma-separated terms (case-insensitive). Compose with sitemapInclude — e.g. include "post" + exclude "tag,category,author" avoids WordPress taxonomy sitemaps (post_tag-sitemap, …-taxonomies-post_tag-*) that also contain "post".

order'newest' | 'oldest' | 'sitemap'

Sort order by sitemap lastmod. "newest" (default) returns the most recent URLs first — the wanted end of a large archive under the limit. URLs without a lastmod sort last.

beforestring

Cursor for paging a large archive newest→oldest: return only URLs with lastmod at or before this ISO timestamp. Page by passing the oldest lastmod from the previous page; dedupe by url across pages.

offsetnumber

Skip the first N results (in the sorted order) then return the next limit. Pagination for sources with no lastmod (WordPress-core), where the date cursor can't apply. Page sequentially (0, limit, 2·limit, …) so the sub-sitemap cache keeps deep pages fast.

includeSubdomainsboolean

Also include URLs on subdomains of the site. Default off.

ignoreQueryParametersboolean

Treat URLs that differ only by query string as one. Default off.

Example request

{
  "url": "https://example.com",
  "includeLinks": true,
  "limit": 1000,
  "search": "pricing",
  "sitemap": "include",
  "sitemapInclude": "post",
  "sitemapExclude": "tag,category,author",
  "before": "2024-06-01T00:00:00Z",
  "offset": 5000
}

Response

The discovered URLs.

Example response

{
  "data": {
    "url": "https://example.com",
    "links": [
      {
        "url": "https://example.com/blog/hello"
      }
    ],
    "count": 2,
    "total": 9051
  }
}

Changes

No recorded changes to this operation across all 1 revision of this API.