Jobs

Create a new crawl job

Creates a new web crawling job with specified parameters. This endpoint crawls websites and extracts content based on configuration.

post/v1/crawl

Request

  • Base URL: https://api.webcrawlerapi.com
  • URL: https://api.webcrawlerapi.com/v1/crawl
  • Auth: API key in header Authorization

Request body

blacklist_regexpstring
items_limitinteger
main_content_onlyboolean
max_ageinteger
max_depthinteger
output_formatsstring[]
respect_robots_txtboolean
urlstring
webhook_urlstring
whitelist_regexpstring

Example request

{
  "blacklist_regexp": ".*\\/admin\\/.*",
  "items_limit": 10,
  "max_age": 604800,
  "max_depth": 2,
  "output_formats": [
    "[\"markdown\"",
    "\"html\"]"
  ],
  "url": "https://example.com",
  "webhook_url": "https://your-domain.com/webhook",
  "whitelist_regexp": ".*\\/blog\\/.*"
}

Response

Job created successfully

object required

Changes

No recorded changes to this operation across all 1 revision of this API.