---
title: "Start Crawl"
method: POST
path: "/site-crawl"
tags: ["site-crawl"]
---

# Start Crawl

`POST /site-crawl`

## Request

- Base URL: `https://api.xeorank.com` (the document declares no server; this is the origin it was published from)
- URL: `https://api.xeorank.com/site-crawl`
- Auth: API key in header `X-Agent-Key`

Create a queued crawl_jobs row. Execution is triggered separately
via Trigger.dev calling POST /site-crawl/{job_id}/_run, OR inline
via BackgroundTasks when ?immediate=1 is set.

## Query parameters

- `immediate` boolean: If true, schedule a FastAPI BackgroundTask to run the crawl in-process. Used by the CLI (long-lived polling keeps the instance warm). The web UI leaves this false and triggers a Trigger.dev task instead.
- `email` string, nullable

## Headers

- `X-Org-Id` string, nullable
- `x-xyle-source` string, nullable

## Request body

- StartSiteCrawlRequest
  - `seed_url` string, required
  - `site_id` string, uuid, nullable
  - `config` SiteCrawlConfig: Public site-crawl config. Several fields are intentionally *not* exposed: - ``user_agent`` cannot be overridden — the crawler always identifies as ``Xyle-Crawler/<version>`` so target sites can identify and block us. - ``respect_robots`` is always true — a public, multi-tenant crawler must never ignore robots.txt. Removed from the surface entirely. Defaults cover a typical brand site while retaining a hard safety ceiling. A crawl that reaches either ceiling is reported as partial rather than silently claiming complete coverage. ``render_js`` defaults to False (Playwright is expensive); enable only when crawling JS-heavy SPAs.
    - `max_pages` integer
    - `max_depth` integer
    - `render_js` boolean
    - `include_subdomains` boolean

## Response `200`

Successful Response

- CrawlJobStatus
  - `job_id` string, required
  - `seed_url` string, required
  - `status` 'queued' | 'running' | 'completed' | 'partial' | 'failed' | 'cancelled', required
  - `stage` string
  - `status_detail` string, nullable
  - `pages_crawled` integer
  - `pages_discovered` integer
  - `counts` CrawlCounts
    - `fetched_pages` integer
    - `recorded_urls` integer
    - `not_fetched_urls` integer
  - `errors_count` integer
  - `health_score` number, nullable
  - `health_breakdown` object, nullable
  - `coverage` object, nullable
  - `error_message` string, nullable
  - `retry_count` integer
  - `last_heartbeat_at` string, date-time, nullable
  - `dispatch_required` boolean
  - `started_at` string, date-time, nullable
  - `finished_at` string, date-time, nullable
  - `created_at` string, date-time, nullable

## Other responses

- `422`: Validation Error

---

[API](https://skmtc.dev/xeorank/apis/seo-intelligence-engine.md) · [All operations](https://skmtc.dev/xeorank/apis/seo-intelligence-engine/llms.txt) · [OpenAPI document](https://skmtc.dev/xeorank/apis/seo-intelligence-engine/revisions/449a3fd73454?raw)