---
title: "Start Full Site Crawl"
method: POST
path: "/settings/scrape-config/crawl"
tags: ["Scraping"]
---

# Start Full Site Crawl

`POST /settings/scrape-config/crawl`

## Request

- Base URL: `https://be.graph8.com/api/v1`
- URL: `https://be.graph8.com/api/v1/settings/scrape-config/crawl`
- Auth: HTTP bearer

Crawl a whole site. RUNS IN THE BACKGROUND and can take a long time.

Returns a job id immediately; poll
`GET /settings/scrape-config/crawl/status/{job_id}` for progress. A full
crawl fetches every reachable page, so point it at a site you own.

## Headers

- `Authorization` string, nullable
- `X-Target-Org-Id` string, nullable

## Request body

- StartCrawlRequest: Crawl a whole site. Long-running, and it fetches every reachable page.
  - `max_pages` integer, nullable
  - `source_url` string, nullable: Where to start. Omit to use the org's configured site.

## Response `202`

Successful Response

- ApiResponseSettingsPayload
  - `data` SettingsPayload, required: An open envelope for a settings object. Open because an ontology concept and an SSO config both grow fields as the product does, and on a settings read a dropped field reads as "not configured" rather than "we did not tell you".
  - `pagination` PaginationMeta
    - `page` integer, required: Current page number (1-indexed)
    - `limit` integer, required: Items per page
    - `total` integer, required: Total number of items
    - `has_next` boolean, required: Whether there are more pages
    - `next_cursor` string, nullable: Opaque cursor for the next page; null when there are no more items. Pass it back as the `cursor` query param to fetch the following page without computing page numbers.
    - `total_available` integer, nullable: Search endpoints only: how many records really match, when known. Can be larger than `total`, which is capped at the 10,000 results that can be paged. For company search this is a distinct-company estimate. Null when unknown; absent outside search.
    - `total_is_lower_bound` boolean, nullable: Search endpoints only: true when more records match than `total` shows (the result set is larger than the 10,000 results that can be paged). Absent outside search.
    - `notes` string[], nullable: Search endpoints only: plain-language notes about the result, e.g. that more records match than can be paged, or that a reversed `between` range was put in order.

## Other responses

- `422`: Validation Error

---

[API](https://skmtc.dev/graph8/apis/graph8-developer-api.md) · [All operations](https://skmtc.dev/graph8/apis/graph8-developer-api/llms.txt) · [OpenAPI document](https://skmtc.dev/graph8/apis/graph8-developer-api/revisions/2e7cfa0ca921?raw)