sitemaps

The site's XML sitemap from a crawl, as signed download links

The site's XML sitemap, generated from a committed crawl, as signed download links valid 15 minutes, one per file. By default: 2xx HTML pages by crawl depth, without noindex, canonicalised or paginated URLs, each with lastmod, changefreq and priority; options widens or narrows it. Past 50,000 URLs or 49,000,000 characters it rolls over into several files plus sitemapindex.xml: file_count says how many, and each file's hosted_url is where it belongs on the site. THIS IS AN EXPORT, NOT A PUBLISH: download the files, commit them to the site so they are served at those addresses, and reference the sitemap (or the index) from robots.txt. Frontshelf never writes to the site.

get/v1/sites/{site}/sitemap

Request

  • Base URL: https://frontshelf.io/api
  • URL: https://frontshelf.io/api/v1/sites/{site}/sitemap
  • Auth: HTTP bearer

Path parameters

sitestring required

Site ID (site_…), origin (https://example.com), or bare domain, resolved server-side; the response carries the canonical ID.

Query parameters

crawl_seqinteger

A committed crawl's sequence number; omit for the site's current accepted crawl.

include_2xxboolean
include_3xxboolean
include_4xxboolean
include_5xxboolean
include_no_responseboolean

Include URLs that never answered (response class 0: a timeout, a DNS failure, a robots refusal).

include_htmlboolean
include_pdfsboolean
include_noindex_pagesboolean
include_canonicalisedboolean
include_paginatedboolean

Include URLs that carry a rel=prev, i.e. pages 2..N of a series. The first page of a series is always included.

include_blocked_by_robotsboolean

Include 2xx pages the crawl marked blocked by robots.txt (only possible when the crawl was configured to ignore and report robots).

include_last_modifiedboolean

<lastmod>: the page's Last-Modified, else the generation date (yyyy-MM-dd, UTC).

include_priorityboolean

<priority>: 1.0 at depth 0 down to 0.5 at depth 5 and beyond.

include_change_frequencyboolean

<changefreq>, read as change_frequency_mode says.

change_frequency_mode'server_response' | 'crawl_depth'

server_response: daily while the page's Last-Modified is under 24 h old, monthly once it is older. crawl_depth: daily for every page.

filename_prefixstring

What the files are called, and so where you host them: sitemap gives sitemap1.xml, sitemap2.xml, ... beside sitemapindex.xml.

Response

The site's XML sitemap from a crawl, as signed download links

summarystring required

The answer in one or two sentences, with the numbers it rests on.

warningsstring[] required

What limits this answer (a cap, a read that was cut short). Empty when nothing does.

crawl_seqinteger required
url_countinteger required

<url> entries across every file.

file_countinteger required

<urlset> files, the index not counted.

content_type'application/xml' required

Changes

No recorded changes to this operation across all 1 revision of this API.