---
title: "Crawl a Site"
method: POST
path: "/crawl"
tags: ["Crawl"]
---

# Crawl a Site

`POST /crawl`

## Request

- Base URL: `https://api.anakin.io/v1`
- URL: `https://api.anakin.io/v1/crawl`
- Auth: API key in header `X-API-Key`

Bulk-fetch markdown across a site — catalog ingestion, site-wide RAG corpora. Submit a URL to crawl. Returns a job ID — poll GET /crawl/{jobId} (with a delay in between) to fetch the completed pages.

## Request body

- CrawlRequest
  - `url` string, uri, required: The starting URL to crawl from.
  - `maxPages` integer: Hard cap on pages fetched.
  - `depth` integer: Link-hops from the starting URL to follow.
  - `country` string: Proxy country code, e.g. us, gb, de.
  - `useBrowser` boolean: Render each page in a headless browser (for SPAs).
  - `includePatterns` string[]: Glob/regex patterns. Only URLs matching at least one pattern are fetched.
  - `excludePatterns` string[]: Glob/regex patterns. URLs matching any pattern are skipped.
  - `sessionId` string: Reuse an existing authenticated browser session (see Browser Sessions) instead of an anonymous request.
  - `sessionName` string: Human-readable name of a saved browser session — alternative to sessionId.

## Response `202`

Job accepted

- CrawlSubmitResponse
  - `jobId` string: The job ID — pass to Get Crawl Result to poll for completion.
  - `status` 'pending' | 'processing'

## Other responses

- `400`: Bad request — missing or malformed body
- `401`: Missing or invalid API key

---

[API](https://skmtc.dev/anakin/apis/anakin-api.md) · [All operations](https://skmtc.dev/anakin/apis/anakin-api/llms.txt) · [OpenAPI document](https://skmtc.dev/anakin/apis/anakin-api/revisions/5eda91e679e5?raw)