urlstring required
The page to start from. An absolute http or https URL.
limitinteger
The most pages to crawl, 1 to 100,000.
maxDepthinteger
How many links deep to follow from the start page. 0 is the start page alone.
maxConcurrencyinteger
How many pages to fetch at once, 1 to 50.
includePathsstring[]
Only crawl URLs whose path and query match one of these regexes, e.g. ^/blog/. Empty means every path.
excludePathsstring[]
Skip URLs whose path and query match any of these regexes. Wins over includePaths.
allowExternalLinksboolean
Follow links onto other domains. Off by default, since with a high limit a crawl can reach many unrelated sites.
allowBackwardLinksboolean
Follow links above the starting path on the same host.
ignoreSitemapboolean
Find pages by following links only, without reading the site's sitemap.
ignoreQueryParametersboolean
Treat URLs that differ only in their query string as the same page.
deduplicateSimilarURLsboolean
Collapse URLs that differ only in tracking parameters or a trailing slash.
delayinteger
Minimum milliseconds between requests to the same host.
respectRobotsboolean
Honor the site's robots.txt for this job.