Integrations

Crawl a website for indexed search

Recursively crawl a website to make it available for indexed search.

get/integrations/web_crawler/index

Request

  • Base URL
    https://api.hyperspell.com
  • URL
    https://api.hyperspell.com/integrations/web_crawler/index
  • Auth
    HTTP bearer

Query parameters

urlstring required

The base URL of the website to crawl

max_depthinteger

Maximum depth of links to follow during crawling

limitinteger

Maximum number of pages to crawl in total

Headers

X-As-Userstring nullable

Optionally set this header to act as a specific user when using an API Key, equivalent to first exchanging the API Key for a User Token

Response

Successful Response

source'reddit' | 'notion' | 'slack' | 'google_calendar' | 'google_mail' | 'imap' | 'google_meet' | 'box' | 'dropbox' | 'github' | 'gitlab' | 'google_drive' | 'vault' | 'web_crawler' | 'trace' | 'microsoft_outlook' | 'microsoft_teams' | 'granola' | 'fathom' | 'fireflies' | 'figma' | 'linear' | 'hubspot' | 'salesforce' | 'coda' | 'confluence' | 'jira' | 'metabase' | 'gong' | 'clickup' | 'lightfield' | 'pylon' | 'fellow' | 'odoo' | 'external_mcp' | 'page' required
resource_idstring required
status'pending' | 'processing' | 'completed' | 'failed' | 'pending_review' | 'skipped' | 'filtered' | 'cancelled' required

Changes