The URLs of a site, from robots.txt sitemaps and the start page's links, optionally ranked by a search (1 credit; free when nothing is found)

Reads the sitemaps listed in robots.txt (or /sitemap.xml), following sitemap indexes; XML, gzip, RSS/Atom and plain-text sitemaps are understood. URLs are cleaned (fragment and tracking parameters such as utm_*, gclid and fbclid dropped), kept on the site, filtered by include_paths / exclude_paths and de-duplicated (www., scheme and trailing slash folded). search ranks the URLs by their path words and titles (BM25) and drops the ones that don't match. Takes at most about 30 seconds.

post/v1/map

Request

  • Base URL: https://api.serpkite.com
  • URL: https://api.serpkite.com/v1/map
  • Auth: HTTP bearer

Request body

urlstring uri required

The site (any page of it; sitemaps are read from its origin)

searchstring

Keep only URLs relevant to these words, most relevant first

limitinteger
include_subdomainsboolean
include_pathsstring[]

Regular expressions matched against the URL path; keep only URLs matching one

exclude_pathsstring[]

Regular expressions matched against the URL path; drop URLs matching one

ignore_query_parametersboolean

Treat URLs that differ only in their query string as one (the first one found is kept)

sitemap'include' | 'only' | 'skip'

include sitemaps and the start page's links, only sitemaps, skip sitemaps (links only)

Response

OK

Changes

No recorded changes to this operation across all 1 revision of this API.