tools

Unified single-fetch, multi-format extraction. One fetch serves every requested format: markdown, html, rawHtml, text, links, metadata. A bot-defence challenge page returns success: false with blocked.vendor whatever its HTTP status; an empty shell or an error placeholder served as HTTP 200 is a failure too. Neither is charged. A PDF or other binary returns 415 UNSUPPORTED_CONTENT_TYPE, unbilled; use process_document. A success whose markdown is empty carries a warning saying so. With escalate: true that same call renders the page once in a stealth browser and returns the formats from it, instead of returning the block.

post/tools/scrape

Request

  • Base URL: https://www.crawlforge.dev/api/v1
  • URL: https://www.crawlforge.dev/api/v1/tools/scrape
  • Auth: API key in header X-API-Key

Request body

urlstring uri required

The URL to scrape

onlyMainContentboolean

Strip navigation, headers, and footers, returning only the main content

respect_robotsboolean
escalateboolean

Retry a blocked page in a stealth browser, in this same call. The plain fetch always runs first; only if it meets a bot wall, an empty shell or an error placeholder does the stealth render run, and the requested formats are then derived from the rendered HTML. The response carries escalated: true|false whenever this is set, plus stealth: { engine, vendor_detected } when the render ran. It costs the escalation add-on only when it ran. The stage needs the CrawlForge execution backend: where that is not configured it returns 503 TOOL_NOT_AVAILABLE and charges nothing. robots.txt is respected on the escalated path too, matched against the same CrawlForge product token.

escalate_engine'playwright' | 'camoufox'

Browser engine for the escalated render (camoufox only where installed on the backend). Ignored when no escalation runs.

max_inline_charsinteger

Largest result returned inline, in characters of its JSON (1000-10000000; env CRAWLFORGE_MAX_INLINE_CHARS sets the default). Over it, the result is stored for 1 hour and the response carries a preview, a result_handle, total_chars and truncated: true; read the rest with read_result (1 credit).

Response

The tool's result; 2 credits charged (see x-credits-note and x-extra for add-ons).

successtrue required
dataobject required

The tool's result. Its shape is tool-specific and documented on the tool's docs page (x-docs-url on the operation).

credits_usedinteger required

Credits charged for this call: the operation's x-credits, plus any add-on x-credits-note or x-extra describes.

credits_remaininginteger required

The account's credit balance after this call.

processing_timeinteger required

Server-side processing time in milliseconds.

warningsstring[]

Non-fatal notes: a robots.txt override that was recorded, a format that could not be produced, a redaction that changed the payload.

Changes

No recorded changes to this operation across all 1 revision of this API.