tools

Pre-built scrapers for popular sites and public APIs — no schema or selectors required. Three modes: name a template and pass a URL; pass template "auto" with a URL to have the matching template picked (the response reports which one it chose); or pass template "list" to enumerate them. Templates come in two kinds, reported as "mode" by the list. An entity template returns one record from one page. A list connector returns N records from one call, as data.items with a data.count — that covers the job-board group (Greenhouse, Lever, Ashby, Workable, Recruitee, Teamtailor), which reads a company's whole board from its ATS API, the government group (NHTSA vPIC VIN decoding, NPPES NPI provider registry), which reads free keyless federal registries, and shopify-collection, which lists a whole storefront collection. A list connector is driven either by a URL or by "params" — e.g. { "template": "greenhouse-jobs", "params": { "company": "stripe" } }. Sites that block plain HTTP fetches (e.g. Amazon) may return sparse data here; the CrawlForge MCP server version uses stealth browsing for those. reddit-thread reads the post from the Arctic Shift archive, because reddit.com blocks direct scraping; the reddit_search tool reads the comment tree. linkedin-profile and tweet are retired — those sites' robots.txt disallow every keyless path — and naming one, or passing one of their URLs to "auto", returns 400 TEMPLATE_UNAVAILABLE with the reason and no charge. Several templates read a machine-readable endpoint rather than the rendered page — shopify-product reads the store's own /products/<handle>.json, npm-package reads the npm registry API, and every list connector reads its platform's API — and report the URL they actually read as "fetched_url". When a Shopify store refuses its JSON endpoint (401, 403, 404 or 410), shopify-product reads the product page's own schema.org JSON-LD instead: the record carries "source": "json-ld" and a warning says so, with per-variant stock, compare-at prices and option names null because JSON-LD does not carry them.

post/tools/scrape_template

Request

  • Base URL: https://www.crawlforge.dev/api/v1
  • URL: https://www.crawlforge.dev/api/v1/tools/scrape_template
  • Auth: API key in header X-API-Key

Request body

OR
OR

Response

The tool's result; 1 credits charged (see x-credits-note and x-extra for add-ons).

successtrue required
dataobject required

The tool's result. Its shape is tool-specific and documented on the tool's docs page (x-docs-url on the operation).

credits_usedinteger required

Credits charged for this call: the operation's x-credits, plus any add-on x-credits-note or x-extra describes.

credits_remaininginteger required

The account's credit balance after this call.

processing_timeinteger required

Server-side processing time in milliseconds.

warningsstring[]

Non-fatal notes: a robots.txt override that was recorded, a format that could not be produced, a redaction that changed the payload.

Changes

No recorded changes to this operation across all 1 revision of this API.