---
title: "Fetch up to 50 URLs concurrently in one synchronous call, returning title, text, and optional CSS-extracted fields per URL"
method: POST
path: "/tools/batch_scrape"
tags: ["tools"]
---

# Fetch up to 50 URLs concurrently in one synchronous call, returning title, text, and optional CSS-extracted fields per URL

`POST /tools/batch_scrape`

## Request

- Base URL: `https://www.crawlforge.dev/api/v1`
- URL: `https://www.crawlforge.dev/api/v1/tools/batch_scrape`
- Auth: API key in header `X-API-Key`

## Request body

- BatchScrapeRequest
  - `urls` object[], required: Array of URLs to scrape (max 50 per synchronous batch on the hosted REST API)
    - `url` string, uri, required: Target URL
    - `id` string: Optional identifier (defaults to the array index)
    - `priority` 'high' | 'medium' | 'low'
    - `custom_config` object
      - `headers` object
      - `timeout` number
      - `selectors` object
  - `respect_robots` boolean: Fetch each origin's robots.txt and skip URLs it disallows for CrawlForge. A missing or unreachable robots.txt is treated as no restrictions. A disallowed URL returns status "skipped" and is not charged; the rest of the batch runs.
  - `batch_config` object: Batch processing configuration
    - `concurrency` number: Max concurrent requests (capped at 8)
    - `delay_between_batches` number
    - `retry_failed` boolean
    - `max_retries` number
    - `fail_fast` boolean
    - `preserve_order` boolean
  - `extraction_template` object: Optional per-page field extraction
    - `fields` object[]: Fields to extract from each page: { name, selector (CSS), attribute? }. Without attribute the element text is returned, one line per block element; xpath is not supported.
      - `name` string, required
      - `selector` string
      - `xpath` string
      - `attribute` string
    - `capture_screenshots` boolean
    - `extract_links` boolean
    - `extract_images` boolean
  - `output_config` object
    - `format` 'json' | 'csv' | 'xml'
    - `include_metadata` boolean
    - `include_errors` boolean
    - `flatten_results` boolean
  - `options` object
    - `user_agent` string
    - `follow_redirects` boolean
    - `javascript_enabled` boolean
    - `respect_robots_txt` boolean
    - `rate_limit_per_domain` number
  - `max_inline_chars` integer: Largest result returned inline, in characters of its JSON (1000-10000000; env CRAWLFORGE_MAX_INLINE_CHARS sets the default). Over it, the result is stored for 1 hour and the response carries a preview, a result_handle, total_chars and truncated: true; read the rest with read_result (1 credit).
  - `redact_pii` union: Remove personal data from the text this call returns, before it is stored or sent back. true is shorthand for { mode: "fast" }: every entity, tagged. As an object: entities (any of EMAIL, PHONE, FINANCIAL, SECRET; omitted or empty means all four, and any other name is a 400 rather than a silent no-op), replace_style ("tag" → <EMAIL>, "mask" → [REDACTED], "remove" → nothing; default "tag") and mode ("fast", the default, is regex-only and costs no extra credits; "model" covers PERSON and LOCATION, needs an LLM and is rejected here — use the CrawlForge MCP server). The response carries redaction: { entities, count, mode } inside data, saying what was removed. Detection is deliberately conservative: a card number must pass Luhn and an IBAN mod-97, so a false positive cannot silently destroy real page content. URLs, queries and identifiers the response uses to name what was fetched are left intact, and counters derived from the text (content_length, word_count, character_count) describe the text as it was extracted, before redaction.
    - boolean
    - object
      - `entities` string[]
      - `replace_style` 'tag' | 'mask' | 'remove'
      - `mode` 'fast' | 'model'

## Response `200`

The tool's result; 5 credits charged (see x-credits-note and x-extra for add-ons).

- ToolSuccess: The envelope every tool returns on success. Credits are deducted only after the tool succeeded, so credits_used is the price of this call and an error response never charges.
  - `success` true, required
  - `data` object, required: The tool's result. Its shape is tool-specific and documented on the tool's docs page (x-docs-url on the operation).
  - `credits_used` integer, required: Credits charged for this call: the operation's x-credits, plus any add-on x-credits-note or x-extra describes.
  - `credits_remaining` integer, required: The account's credit balance after this call.
  - `processing_time` integer, required: Server-side processing time in milliseconds.
  - `warnings` string[]: Non-fatal notes: a robots.txt override that was recorded, a format that could not be produced, a redaction that changed the payload.

## Other responses

- `400`: VALIDATION_ERROR: the body does not match the request schema.
- `401`: MISSING_API_KEY or INVALID_API_KEY.
- `402`: INSUFFICIENT_CREDITS: the balance does not cover the tool's base price. X-Auto-Recharge-Triggered says whether an automatic recharge was started; retry once it lands.
- `429`: RATE_LIMIT_EXCEEDED: your plan's rate limit. Retry-After: 60.
- `500`: TOOL_ERROR, INTERNAL_ERROR or CREDIT_DEDUCTION_FAILED.
- `default`: Any other error. Which of 403, 404, 413, 422, 501, 502, 503 and 504 a tool can return depends on what it does (fetching a page, calling the execution backend, reading stored results); the code vocabulary is listed on the ErrorResponse schema.

---

[API](https://skmtc.dev/crawlforge/apis/crawlforge-rest-api.md) · [All operations](https://skmtc.dev/crawlforge/apis/crawlforge-rest-api/llms.txt) · [OpenAPI document](https://skmtc.dev/crawlforge/apis/crawlforge-rest-api/revisions/a5532fa8d5d5?raw)