Crawler

Create config

Create a new crawling configuration

post/v4/crawl/config

Request

Base URL: https://api.kadoa.com
URL: https://api.kadoa.com/v4/crawl/config
Auth: API key in header x-api-key

Request body

urlstring uri

Single URL to start crawling (for backward compatibility)

startUrlsstring[]

List of URLs for crawling

pathsFilterInstring[]

Regex patterns to include specific full URLs. Accepts array or JSON-stringified array of regex source strings.

pathsFilterOutstring[]

Regex patterns to exclude specific full URLs. Accepts array or JSON-stringified array of regex source strings.

proxyTypestring nullable

Type of proxy to use

proxyCountrystring nullable

Country for proxy selection

timeoutnumber

Timeout in milliseconds

maxDepthnumber

Maximum crawling depth

maxPagesnumber

Maximum number of pages to crawl

maxMatchesnumber

Maximum number of matched pages to crawl before stopping

concurrencynumber

Number of concurrent crawlers

strictDomainboolean

Whether to stay within the same domain

loadImagesboolean

Whether to load images during crawling

safeModeboolean

Enable safe mode for crawling

callbackUrlstring uri nullable

Webhook URL for completion notifications

processDuringCrawlboolean

Whether to run preprocessing and extraction during the crawl phase

matchThresholdnumber

Match threshold override for blueprint filtering

rawModeboolean

Whether this is a raw data mode crawl

jobIdstring

Internal: Job ID for workflow tracking

workflowIdstring

Internal: Owning workflow (scraperId) for BrightData cost attribution

dataKeystring

Internal: Data key for Parquet storage path

billingSourcestring

Internal: Billing source identifier

Response

200