tools

Crawl websites deeply using breadth-first search to discover and extract content

post/tools/crawl_deep

Request

  • Base URL: https://www.crawlforge.dev/api/v1
  • URL: https://www.crawlforge.dev/api/v1/tools/crawl_deep
  • Auth: API key in header X-API-Key

Request body

start_urlstring uri required

The starting URL for the crawl

max_pagesnumber

Maximum number of pages to crawl (1-100)

max_depthnumber

Maximum depth to crawl (1-5)

same_domain_onlyboolean

Only crawl pages on the same domain

respect_robotsboolean

Fetch each origin's robots.txt and skip URLs it disallows for CrawlForge. A missing or unreachable robots.txt is treated as no restrictions. Returns 403 ROBOTS_DISALLOWED when start_url itself is disallowed, with no credits charged; URLs disallowed mid-crawl are skipped and counted in warnings. Setting this to false is recorded against your API key.

respect_robots_txtboolean

Alias of respect_robots, kept for callers that already send it. Either parameter set to false disables the gate.

crawl_delaynumber

Delay between requests to the same host, in milliseconds (0-5000). Resolved against the site's own robots.txt Crawl-delay as the longer of the two, so the site wins when it asks for more and the two never stack. A delay that will not fit in the crawl timeout skips the page rather than overrunning the budget.

timeoutnumber

Total crawl timeout in milliseconds (1000-60000)

max_inline_charsinteger

Largest result returned inline, in characters of its JSON (1000-10000000; env CRAWLFORGE_MAX_INLINE_CHARS sets the default). Over it, the result is stored for 1 hour and the response carries a preview, a result_handle, total_chars and truncated: true; read the rest with read_result (1 credit).

Response

The tool's result; 4 credits charged (see x-credits-note and x-extra for add-ons).

successtrue required
dataobject required

The tool's result. Its shape is tool-specific and documented on the tool's docs page (x-docs-url on the operation).

credits_usedinteger required

Credits charged for this call: the operation's x-credits, plus any add-on x-credits-note or x-extra describes.

credits_remaininginteger required

The account's credit balance after this call.

processing_timeinteger required

Server-side processing time in milliseconds.

warningsstring[]

Non-fatal notes: a robots.txt override that was recorded, a format that could not be produced, a redaction that changed the payload.

Changes

No recorded changes to this operation across all 1 revision of this API.