crawler

Start Crawl

Start a new crawl job using Celery.

The crawl runs in a Celery worker and can be monitored via the status endpoint.

post/api/crawl/start

Request

  • Base URL: https://voyant.io (the document declares no server; this is the origin it was published from)
  • URL: https://voyant.io/api/crawl/start
  • Auth: HTTP bearer

Request body

domainsstring[] required

List of domains to crawl

max_pagesinteger

Maximum pages to crawl

max_depthinteger

Maximum crawl depth

respect_robotsboolean

Respect robots.txt

same_domain_onlyboolean

Only crawl same domain

static_workersinteger

Parallel workers for static sites

headlessboolean

Run browser in headless mode

include_patternsstring[]

Regex URL/path patterns to include

exclude_patternsstring[]

Regex URL/path patterns to exclude

exclude_subdomainsstring[]

Subdomains to exclude (e.g., ['docs', 'api', 'cdn'])

Response

Successful Response

crawl_idstring required
task_idstring required
statusstring required
messagestring required

Changes

No recorded changes to this operation across all 1 revision of this API.