Scraping

Start Full Site Crawl

Crawl a whole site. RUNS IN THE BACKGROUND and can take a long time.

Returns a job id immediately; poll GET /settings/scrape-config/crawl/status/{job_id} for progress. A full crawl fetches every reachable page, so point it at a site you own.

post/settings/scrape-config/crawl

Request

  • Base URL: https://be.graph8.com/api/v1
  • URL: https://be.graph8.com/api/v1/settings/scrape-config/crawl
  • Auth: HTTP bearer

Headers

Authorizationstring nullable
X-Target-Org-Idstring nullable

Request body

max_pagesinteger nullable
source_urlstring nullable

Where to start. Omit to use the org's configured site.

Response

Successful Response

dataSettingsPayload required

An open envelope for a settings object.

Open because an ontology concept and an SSO config both grow fields as the product does, and on a settings read a dropped field reads as "not configured" rather than "we did not tell you".

Changes

No recorded changes to this operation across all 2 revisions of this API.