V2 Scraping

Create V2 scrape job

Creates a new V2 scrape job with improved output formats and optional async mode. Returns content directly in sync mode or job ID in async mode.

post/v2/scrape

Request

  • Base URL: https://api.webcrawlerapi.com
  • URL: https://api.webcrawlerapi.com/v2/scrape
  • Auth: API key in header Authorization

Query parameters

asyncboolean

Run asynchronously (returns job ID instead of waiting)

Request body

clean_selectorsstring
main_content_onlyboolean
max_ageinteger
output_formatstring

Deprecated: use output_formats instead.

output_formatsstring[]
promptstring
respect_robots_txtboolean
response_schemaobject
urlstring

Example request

{
  "clean_selectors": ".ad,.sidebar",
  "max_age": 604800,
  "output_format": "markdown",
  "output_formats": [
    "[\"markdown\"",
    "\"html\"]"
  ],
  "prompt": "Extract all product prices",
  "url": "https://example.com"
}

Response

Job ID (async mode)

object required

Changes

No recorded changes to this operation across all 1 revision of this API.