tools

Fetch a document by URL and extract its real text, metadata, and tables. Supported on this endpoint: PDF (text, metadata, tables via pdf-parse), CSV (text + parsed rows), TXT, and HTML (text + metadata). DOCX/XLSX return 501 and are supported by the CrawlForge MCP server.

post/tools/process_document

Request

  • Base URL: https://www.crawlforge.dev/api/v1
  • URL: https://www.crawlforge.dev/api/v1/tools/process_document
  • Auth: API key in header X-API-Key

Request body

urlstring uri required

URL of the document to process (max 25MB)

respect_robotsboolean

Fetch the origin's robots.txt and refuse the document if it disallows CrawlForge. A missing or unreachable robots.txt is treated as no restrictions. Returns 403 ROBOTS_DISALLOWED, and no credits are charged.

document_type'pdf' | 'docx' | 'xlsx' | 'csv' | 'txt' | 'auto'

Document type; auto-detected from Content-Type, URL extension, then content sniffing

extract_textboolean

Extract text content (capped at 200000 characters; PDFs read up to 200 pages, and pages_read { start, end, count, total_pages } says which; HTML is read one line per block element)

extract_metadataboolean

Extract document metadata (PDF info dictionary; HTML title/description/author)

extract_tablesboolean

Extract tables (PDF via ruled-grid detection, CSV via parsing; not available for HTML). Up to 20 tables of 1000 rows; a table ruled only horizontally keeps its rows but reports one column

extract_imagesboolean

Not available on the hosted REST API — returns images: null with a note; use the CrawlForge MCP server for image extraction

timeoutnumber

Fetch timeout in milliseconds (1000-60000; capped at 20000 to fit the serverless limit)

max_inline_charsinteger

Largest result returned inline, in characters of its JSON (1000-10000000; env CRAWLFORGE_MAX_INLINE_CHARS sets the default). Over it, the result is stored for 1 hour and the response carries a preview, a result_handle, total_chars and truncated: true; read the rest with read_result (1 credit).

Response

The tool's result; 2 credits charged (see x-credits-note and x-extra for add-ons).

successtrue required
dataobject required

The tool's result. Its shape is tool-specific and documented on the tool's docs page (x-docs-url on the operation).

credits_usedinteger required

Credits charged for this call: the operation's x-credits, plus any add-on x-credits-note or x-extra describes.

credits_remaininginteger required

The account's credit balance after this call.

processing_timeinteger required

Server-side processing time in milliseconds.

warningsstring[]

Non-fatal notes: a robots.txt override that was recorded, a format that could not be produced, a redaction that changed the payload.

Changes

No recorded changes to this operation across all 1 revision of this API.