Sign in
Crawl

Create a website datasource

Create a new website datasource and optionally start an initial crawl. Crawled content is automatically indexed into your knowledge base.

post/crawl

Request body

urlstring uri required
display_namestring
page_limitinteger nullable
exclude_pathsstring[]
include_pathsstring[]
crawl_interval_hoursinteger nullable
auto_start_crawlboolean
restricted_to_channelsSessionChannel[]

Response

Default Response

idstring uuid required
urlstring required
display_namestring required
statusstring required
page_limitnumber nullable required
exclude_pathsstring[] required
include_pathsstring[] required
crawl_interval_hoursnumber nullable required
restricted_to_channelsSessionChannel[] required

Channels this datasource is restricted to. Empty = available on all channels.

last_crawl_started_atstring date-time nullable required
last_crawl_completed_atstring date-time nullable required
error_messagestring nullable required
created_atstring date-time required
updated_atstring date-time required
auth_configuredboolean required

Whether optional Firecrawl site auth is configured. Secrets are never returned.

auth_type'basic' | 'bearer' | 'cookie' | 'headers' nullable required

Configured auth type, or null when auth is not set.

Changes