Web: Map

Map a website's URLs

Returns a list of URLs found on a website, quickly. Useful for discovering the structure of a site before scraping specific pages. Optionally order results by relevance to a search term.

post/v1/data/web/map

Request

  • Base URL: https://api.superagnt.com
  • URL: https://api.superagnt.com/v1/data/web/map
  • Auth: API key in header Authorization

Request body

urlstring uri required

The base URL of the site to map.

searchstring

Order returned URLs by relevance to this query.

sitemap'skip' | 'include' | 'only'

How to use the site's sitemap. Default 'include'.

includeSubdomainsboolean

Include URLs on subdomains.

ignoreQueryParametersboolean

Treat URLs that differ only by query string as the same URL.

limitinteger

Maximum number of URLs to return. Default 5000, max 100000.

timeoutinteger

Request timeout in milliseconds.

Response

The list of URLs discovered on the site.

object required

Changes

No recorded changes to this operation across all 1 revision of this API.