Sign in
crawling

Create Crawl Graph

Create a crawl graph by recursively crawling from a root URL

This endpoint crawls a website starting from the given URL up to the specified depth, extracting links and building a graph structure. Results are checkpointed to S3.

Requires permission: crawl:write

post/crawl/graph

Request body

urlstring uri required

Root URL to start crawling from

enginesobject[] required

List of engines to use

crawl_idstring required

Unique identifier for this crawl

s3_bucketstring required

S3 bucket for checkpointing

depthinteger nullable

Maximum crawl depth

absolute_onlyboolean

Only extract absolute URLs

keep_externalboolean

Keep external URLs in results

visit_externalboolean

Visit external URLs

max_workersinteger

Maximum concurrent workers

batch_sizeinteger

URLs per batch

max_urlsinteger

Maximum URLs to crawl

debugboolean

Enable debug information

Example request

{
  "crawl_id": "crawl-123",
  "depth": 3,
  "engines": [
    {
      "engine_type": "scrapingbee"
    }
  ],
  "max_urls": 500,
  "s3_bucket": "my-crawl-bucket",
  "url": "https://example.com"
}

Response

Successful Response

crawl_idstring required
root_urlstring required
total_pages_visitedinteger required
visited_urlsstring[] required
depth_reachedinteger required
max_depthinteger required
max_urlsinteger required
absolute_onlyboolean required
keep_externalboolean required
visit_externalboolean required
max_workersinteger required
batch_sizeinteger required
timestampstring required
debug_frameobject[] nullable

Changes