Sign in
crawling

Create Crawl Graph

Create a crawl graph by recursively crawling from a root URL

This endpoint crawls a website starting from the given URL up to the specified depth, extracting links and building a graph structure. Results are checkpointed.

Requires permission: crawl:write

post/crawl/graph

Request body

urlstring uri required

Root URL to start crawling from

enginesEngineType[] required

List of engines to use

crawl_idstring required

Unique identifier for this crawl

depthinteger nullable

Maximum crawl depth

absolute_onlyboolean

Only extract absolute URLs

keep_externalboolean

Keep external URLs in results

visit_externalboolean

Visit external URLs

max_workersinteger

Maximum concurrent workers

batch_sizeinteger

URLs per batch

max_urlsinteger

Maximum URLs to crawl

debugboolean

Enable debug information

Response

Successful Response

crawl_idstring required
root_urlstring required
total_pages_visitedinteger required
visited_urlsstring[] required
depth_reachedinteger required
max_depthinteger required
max_urlsinteger required
absolute_onlyboolean required
keep_externalboolean required
visit_externalboolean required
max_workersinteger required
batch_sizeinteger required
timestampstring required
debug_frameobject[] nullable

Changes

Changed in 3 of the 16 revisions of this API.218

  • 29225d7b3ea412See the full diff
    • ●

      removed the request property s3_bucket

      request-property-removed

    • ○

      added the optional property detail/items/ctx to the response with the 422 status

      response-optional-property-added

    • ○

      added the optional property detail/items/input to the response with the 422 status

      response-optional-property-added

  • 019c4555bd7f1See the full diff
    • ○

      added the new ASYNC_FLEET_STICKY enum value to the request property engines/items/

      request-property-enum-value-added

  • 707c0eb9cee125See the full diff
    • ▲

      request property engines/items/ was restricted to a list of enum values

      request-property-became-enum

    • ▲

      the engines/items/ request property type/format changed from object/ to string/

      request-property-type-changed

    • ○

      added the new FLEET enum value to the request property engines/items/

      request-property-enum-value-added

    • ○

      added the new FLEET_ASYNC enum value to the request property engines/items/

      request-property-enum-value-added

    • ○

      added the new FLEET_WORKFLOW enum value to the request property engines/items/

      request-property-enum-value-added

    • ○

      added the new SCRAPINGBEE enum value to the request property engines/items/

      request-property-enum-value-added

    • ○

      added the new ZENROWS enum value to the request property engines/items/

      request-property-enum-value-added