tools
Pre-built scrapers for popular sites and public APIs — no schema or selectors required. Three modes: name a template and pass a URL; pass template "auto" with a URL to have the matching template picked (the response reports which one it chose); or pass template "list" to enumerate them. Templates come in two kinds, reported as "mode" by the list. An entity template returns one record from one page. A list connector returns N records from one call, as data.items with a data.count — that covers the job-board group (Greenhouse, Lever, Ashby, Workable, Recruitee, Teamtailor), which reads a company's whole board from its ATS API, the government group (NHTSA vPIC VIN decoding, NPPES NPI provider registry), which reads free keyless federal registries, and shopify-collection, which lists a whole storefront collection. A list connector is driven either by a URL or by "params" — e.g. { "template": "greenhouse-jobs", "params": { "company": "stripe" } }. Sites that block plain HTTP fetches (e.g. Amazon) may return sparse data here; the CrawlForge MCP server version uses stealth browsing for those. reddit-thread reads the post from the Arctic Shift archive, because reddit.com blocks direct scraping; the reddit_search tool reads the comment tree. linkedin-profile and tweet are retired — those sites' robots.txt disallow every keyless path — and naming one, or passing one of their URLs to "auto", returns 400 TEMPLATE_UNAVAILABLE with the reason and no charge. Several templates read a machine-readable endpoint rather than the rendered page — shopify-product reads the store's own /products/<handle>.json, npm-package reads the npm registry API, and every list connector reads its platform's API — and report the URL they actually read as "fetched_url". When a Shopify store refuses its JSON endpoint (401, 403, 404 or 410), shopify-product reads the product page's own schema.org JSON-LD instead: the record carries "source": "json-ld" and a warning says so, with per-variant stock, compare-at prices and option names null because JSON-LD does not carry them.
post/tools/scrape_template
Request
- Base URL: https://www.crawlforge.dev/api/v1
- URL: https://www.crawlforge.dev/api/v1/tools/scrape_template
- Auth: API key in header X-API-Key
Request body
Response
The tool's result; 1 credits charged (see x-credits-note and x-extra for add-ons).
Changes
No recorded changes to this operation across all 1 revision of this API.