Crawl

Crawl a Site

Bulk-fetch markdown across a site — catalog ingestion, site-wide RAG corpora. Submit a URL to crawl. Returns a job ID — poll GET /crawl/{jobId} (with a delay in between) to fetch the completed pages.

post/crawl

Request

  • Base URL: https://api.anakin.io/v1
  • URL: https://api.anakin.io/v1/crawl
  • Auth: API key in header X-API-Key

Request body

urlstring uri required

The starting URL to crawl from.

maxPagesinteger

Hard cap on pages fetched.

depthinteger

Link-hops from the starting URL to follow.

countrystring

Proxy country code, e.g. us, gb, de.

useBrowserboolean

Render each page in a headless browser (for SPAs).

includePatternsstring[]

Glob/regex patterns. Only URLs matching at least one pattern are fetched.

excludePatternsstring[]

Glob/regex patterns. URLs matching any pattern are skipped.

sessionIdstring

Reuse an existing authenticated browser session (see Browser Sessions) instead of an anonymous request.

sessionNamestring

Human-readable name of a saved browser session — alternative to sessionId.

Example request

{
  "url": "https://example.com"
}

Response

Job accepted

jobIdstring

The job ID — pass to Get Crawl Result to poll for completion.

status'pending' | 'processing'

Changes

No recorded changes to this operation across all 1 revision of this API.