--- title: "Pre-built scrapers for popular sites and public APIs — no schema or selectors required. Three modes: name a template and pass a URL; pass template \"auto\" with a URL to have the matching template picked (the response reports which one it chose); or pass template \"list\" to enumerate them. Templates come in two kinds, reported as \"mode\" by the list. An entity template returns one record from one page. A list connector returns N records from one call, as data.items with a data.count — that covers the job-board group (Greenhouse, Lever, Ashby, Workable, Recruitee, Teamtailor), which reads a company's whole board from its ATS API, the government group (NHTSA vPIC VIN decoding, NPPES NPI provider registry), which reads free keyless federal registries, and shopify-collection, which lists a whole storefront collection. A list connector is driven either by a URL or by \"params\" — e.g. { \"template\": \"greenhouse-jobs\", \"params\": { \"company\": \"stripe\" } }. Sites that block plain HTTP fetches (e.g. Amazon) may return sparse data here; the CrawlForge MCP server version uses stealth browsing for those. reddit-thread reads the post from the Arctic Shift archive, because reddit.com blocks direct scraping; the reddit_search tool reads the comment tree. linkedin-profile and tweet are retired — those sites' robots.txt disallow every keyless path — and naming one, or passing one of their URLs to \"auto\", returns 400 TEMPLATE_UNAVAILABLE with the reason and no charge. Several templates read a machine-readable endpoint rather than the rendered page — shopify-product reads the store's own /products/.json, npm-package reads the npm registry API, and every list connector reads its platform's API — and report the URL they actually read as \"fetched_url\". When a Shopify store refuses its JSON endpoint (401, 403, 404 or 410), shopify-product reads the product page's own schema.org JSON-LD instead: the record carries \"source\": \"json-ld\" and a warning says so, with per-variant stock, compare-at prices and option names null because JSON-LD does not carry them." method: POST path: "/tools/scrape_template" tags: ["tools"] --- # Pre-built scrapers for popular sites and public APIs — no schema or selectors required. Three modes: name a template and pass a URL; pass template "auto" with a URL to have the matching template picked (the response reports which one it chose); or pass template "list" to enumerate them. Templates come in two kinds, reported as "mode" by the list. An entity template returns one record from one page. A list connector returns N records from one call, as data.items with a data.count — that covers the job-board group (Greenhouse, Lever, Ashby, Workable, Recruitee, Teamtailor), which reads a company's whole board from its ATS API, the government group (NHTSA vPIC VIN decoding, NPPES NPI provider registry), which reads free keyless federal registries, and shopify-collection, which lists a whole storefront collection. A list connector is driven either by a URL or by "params" — e.g. { "template": "greenhouse-jobs", "params": { "company": "stripe" } }. Sites that block plain HTTP fetches (e.g. Amazon) may return sparse data here; the CrawlForge MCP server version uses stealth browsing for those. reddit-thread reads the post from the Arctic Shift archive, because reddit.com blocks direct scraping; the reddit_search tool reads the comment tree. linkedin-profile and tweet are retired — those sites' robots.txt disallow every keyless path — and naming one, or passing one of their URLs to "auto", returns 400 TEMPLATE_UNAVAILABLE with the reason and no charge. Several templates read a machine-readable endpoint rather than the rendered page — shopify-product reads the store's own /products/.json, npm-package reads the npm registry API, and every list connector reads its platform's API — and report the URL they actually read as "fetched_url". When a Shopify store refuses its JSON endpoint (401, 403, 404 or 410), shopify-product reads the product page's own schema.org JSON-LD instead: the record carries "source": "json-ld" and a warning says so, with per-variant stock, compare-at prices and option names null because JSON-LD does not carry them. `POST /tools/scrape_template` ## Request - Base URL: `https://www.crawlforge.dev/api/v1` - URL: `https://www.crawlforge.dev/api/v1/tools/scrape_template` - Auth: API key in header `X-API-Key` ## Request body - union - object - `template` 'list', required: A supported template id, "auto" to pick one from the url, or "list" to discover them. - `url` string, uri, required: URL to scrape. Required unless template === "list", or params drive a list connector. Always required for template "auto". - `params` object: Drives a list connector in place of a url — { "company": "stripe" } for the job boards, { "store": "www.allbirds.com", "collection": "mens" } for shopify-collection, { "vin": "5UXWX7C5*BA" } for nhtsa-vin. Each connector names its own parameters in the description that template "list" returns. A missing or invalid parameter is a 400 naming it, and no credits are charged. - `respect_robots` boolean: Fetch the origin's robots.txt and refuse the URL if it disallows CrawlForge. Checked against the URL actually fetched, which is the API endpoint rather than the URL or params you pass wherever a template resolves one. A missing or unreachable robots.txt is treated as no restrictions. Returns 403 ROBOTS_DISALLOWED, and no credits are charged. - `timeout` integer: Request timeout in milliseconds (5000–60000). - `max_inline_chars` integer: Largest result returned inline, in characters of its JSON (1000-10000000; env CRAWLFORGE_MAX_INLINE_CHARS sets the default). Over it, the result is stored for 1 hour and the response carries a preview, a result_handle, total_chars and truncated: true; read the rest with read_result (1 credit). - object - `template` string, required: A supported template id, "auto" to pick one from the url, or "list" to discover them. - `url` string, uri, required: URL to scrape. Required unless template === "list", or params drive a list connector. Always required for template "auto". - `params` object: Drives a list connector in place of a url — { "company": "stripe" } for the job boards, { "store": "www.allbirds.com", "collection": "mens" } for shopify-collection, { "vin": "5UXWX7C5*BA" } for nhtsa-vin. Each connector names its own parameters in the description that template "list" returns. A missing or invalid parameter is a 400 naming it, and no credits are charged. - `respect_robots` boolean: Fetch the origin's robots.txt and refuse the URL if it disallows CrawlForge. Checked against the URL actually fetched, which is the API endpoint rather than the URL or params you pass wherever a template resolves one. A missing or unreachable robots.txt is treated as no restrictions. Returns 403 ROBOTS_DISALLOWED, and no credits are charged. - `timeout` integer: Request timeout in milliseconds (5000–60000). - `max_inline_chars` integer: Largest result returned inline, in characters of its JSON (1000-10000000; env CRAWLFORGE_MAX_INLINE_CHARS sets the default). Over it, the result is stored for 1 hour and the response carries a preview, a result_handle, total_chars and truncated: true; read the rest with read_result (1 credit). - object - `template` string, required: A supported template id, "auto" to pick one from the url, or "list" to discover them. - `url` string, uri, required: URL to scrape. Required unless template === "list", or params drive a list connector. Always required for template "auto". - `params` object, required: Drives a list connector in place of a url — { "company": "stripe" } for the job boards, { "store": "www.allbirds.com", "collection": "mens" } for shopify-collection, { "vin": "5UXWX7C5*BA" } for nhtsa-vin. Each connector names its own parameters in the description that template "list" returns. A missing or invalid parameter is a 400 naming it, and no credits are charged. - `respect_robots` boolean: Fetch the origin's robots.txt and refuse the URL if it disallows CrawlForge. Checked against the URL actually fetched, which is the API endpoint rather than the URL or params you pass wherever a template resolves one. A missing or unreachable robots.txt is treated as no restrictions. Returns 403 ROBOTS_DISALLOWED, and no credits are charged. - `timeout` integer: Request timeout in milliseconds (5000–60000). - `max_inline_chars` integer: Largest result returned inline, in characters of its JSON (1000-10000000; env CRAWLFORGE_MAX_INLINE_CHARS sets the default). Over it, the result is stored for 1 hour and the response carries a preview, a result_handle, total_chars and truncated: true; read the rest with read_result (1 credit). ## Response `200` The tool's result; 1 credits charged (see x-credits-note and x-extra for add-ons). - ToolSuccess: The envelope every tool returns on success. Credits are deducted only after the tool succeeded, so credits_used is the price of this call and an error response never charges. - `success` true, required - `data` object, required: The tool's result. Its shape is tool-specific and documented on the tool's docs page (x-docs-url on the operation). - `credits_used` integer, required: Credits charged for this call: the operation's x-credits, plus any add-on x-credits-note or x-extra describes. - `credits_remaining` integer, required: The account's credit balance after this call. - `processing_time` integer, required: Server-side processing time in milliseconds. - `warnings` string[]: Non-fatal notes: a robots.txt override that was recorded, a format that could not be produced, a redaction that changed the payload. ## Other responses - `400`: VALIDATION_ERROR: the body does not match the request schema. - `401`: MISSING_API_KEY or INVALID_API_KEY. - `402`: INSUFFICIENT_CREDITS: the balance does not cover the tool's base price. X-Auto-Recharge-Triggered says whether an automatic recharge was started; retry once it lands. - `429`: RATE_LIMIT_EXCEEDED: your plan's rate limit. Retry-After: 60. - `500`: TOOL_ERROR, INTERNAL_ERROR or CREDIT_DEDUCTION_FAILED. - `default`: Any other error. Which of 403, 404, 413, 422, 501, 502, 503 and 504 a tool can return depends on what it does (fetching a page, calling the execution backend, reading stored results); the code vocabulary is listed on the ErrorResponse schema. --- [API](https://skmtc.dev/crawlforge/apis/crawlforge-rest-api.md) · [All operations](https://skmtc.dev/crawlforge/apis/crawlforge-rest-api/llms.txt) · [OpenAPI document](https://skmtc.dev/crawlforge/apis/crawlforge-rest-api/revisions/a5532fa8d5d5?raw)