Sign in
Conversational AI

Create Crawl Job

Create a crawl job to crawl the given URL with specified depth and page limits.

post/v1/convai/knowledge-base/crawl

Headers

xi-api-keystring nullable

Your API key. This is required by most endpoints to access our API programmatically. You can view your xi-api-key using the 'Profile' tab on the website.

Your API key. This is required by most endpoints to access our API programmatically. You can view your xi-api-key using the 'Profile' tab on the website.

Request body

urlstring required

URL to a page of documentation that the agent will have access to in order to interact with users.

max_depthinteger

Deprecated - this field is a no-op and will be removed in a future version.

max_pagesinteger

Maximum number of pages to crawl (1-10,000), defaults to 1000.

patternstring nullable

If set, only URLs that match this pattern are included.

sitemap_urlsstring[] nullable

List of URLs to crawl from sitemap (optional, overrides automatic URL discovery).

parent_folder_idstring nullable

If set, the created document or folder will be placed inside the given folder.

enable_auto_syncboolean

Whether to enable auto-sync for this URL document.

auto_removeboolean

Whether to automatically remove the document if the URL becomes unavailable. Only applicable when auto-sync is enabled.

auto_discoverboolean

Automatically discover and add new pages linked from already-crawled pages during auto-sync. Requires enable_auto_sync=true.

minimum_frequency_daysinteger nullable

Minimum frequency (in days) at which the underlying eligible documents are refreshed. The actual interval may be shorter, never longer. Defaults to 7, tightened to the parent folder's frequency if that is stricter. Only applicable when auto-sync is enabled.

Response

Successful Response

idstring required
type'discovery' | 'sitemap' required
root_folder_idstring required
statusstring required
created_atinteger required

Changes

Changed in 4 of the 49 revisions of this API.4