Sign in
crawl-settings

Get web crawl setting

Gets the setting (schedule, URL, etc.) for web crawl data sources.

Required scope: `ingestion:manage`

get/crawl-settings/{id}

Path parameters

idstring required

Ingestion setting ID for the web crawl data source

Response

Successful retrieval of web crawl setting

idstring required

Web crawl ingestion setting ID

namestring required

Name of the web crawl ingestion setting

start_urlstring required

Start URL of the web crawl

path_filtersstring[]

Path filters for crawling

content_patternstring[]

Content patterns for filtering

max_depthinteger required

Maximum crawl depth

max_num_filesinteger nullable

Maximum number of files to crawl

html_onlyboolean required

When true, only HTML files will be downloaded

use_browserboolean required

Whether to use a headless browser for crawling

no_canonical_checkboolean

Whether to disable duplicate filtering by canonical URL

file_extensionsstring[]

File extensions to include (e.g. ".pdf", ".docx")

rrulestring

Recurrence rule (RFC 5545 RRULE)

Changes

Changed in 1 of the 71 revisions of this API.1

Of the 71 revisions, 18 have no diff computed.