tools

Enumerate a site's pages: reads {origin}/sitemap.xml when available (up to 500 URLs, following up to 3 child sitemaps), otherwise falls back to a bounded breadth-first crawl (up to 30 pages, ~18s budget)

post/tools/map_site

Request

  • Base URL: https://www.crawlforge.dev/api/v1
  • URL: https://www.crawlforge.dev/api/v1/tools/map_site
  • Auth: API key in header X-API-Key

Request body

urlstring uri required

The starting URL to map

respect_robotsboolean

Fetch each origin's robots.txt and skip URLs it disallows for CrawlForge. A missing or unreachable robots.txt is treated as no restrictions. Returns 403 ROBOTS_DISALLOWED when the start URL itself is disallowed, with no credits charged; a disallowed sitemap.xml falls back to the crawl, and pages disallowed mid-crawl are skipped and counted in warnings.

max_depthnumber

Maximum crawl depth for the fallback crawl (1-5); unused when a sitemap is found

include_externalboolean

Include external links in per-page link lists (crawl mode only; external pages are never crawled)

timeoutnumber

Overall crawl budget in milliseconds (1000-30000; capped at ~18000 to fit the serverless limit)

max_inline_charsinteger

Largest result returned inline, in characters of its JSON (1000-10000000; env CRAWLFORGE_MAX_INLINE_CHARS sets the default). Over it, the result is stored for 1 hour and the response carries a preview, a result_handle, total_chars and truncated: true; read the rest with read_result (1 credit).

Response

The tool's result; 2 credits charged (see x-credits-note and x-extra for add-ons).

successtrue required
dataobject required

The tool's result. Its shape is tool-specific and documented on the tool's docs page (x-docs-url on the operation).

credits_usedinteger required

Credits charged for this call: the operation's x-credits, plus any add-on x-credits-note or x-extra describes.

credits_remaininginteger required

The account's credit balance after this call.

processing_timeinteger required

Server-side processing time in milliseconds.

warningsstring[]

Non-fatal notes: a robots.txt override that was recorded, a format that could not be produced, a redaction that changed the payload.

Changes

No recorded changes to this operation across all 1 revision of this API.