Sign in
crawls

Create a web crawl data source

Creates a new web crawl data source and starts ingestion.

Required scope: `ingestion:manage`

post/crawls

Request body

namestring required

Name of the web crawl data source

start_urlstring required

Start URL of the web crawl

path_filtersstring[]

Path filters for crawling. The total number of characters across all elements in the array must be 2000 or fewer.

content_patternstring[]

Content patterns for filtering. The total number of characters across all elements in the array must be 2000 or fewer.

max_depthinteger required

Maximum crawl depth

max_num_filesinteger required

Maximum number of files to crawl

html_onlyboolean

When true, only HTML files will be downloaded

use_browserboolean

Whether to use a headless browser for crawling

no_canonical_checkboolean

Whether to disable duplicate filtering by canonical URL

file_extensionsstring[]

File extensions to include (e.g. ".pdf", ".docx"). For supported file extensions, please refer to https://developer.qaip.com/docs/datasources#%E5%AF%BE%E5%BF%9C%E3%81%97%E3%81%A6%E3%81%84%E3%82%8B%E3%83%95%E3%82%A1%E3%82%A4%E3%83%AB%E5%BD%A2%E5%BC%8F

rrulestring

Recurrence rule (RFC 5545 RRULE)

Response

Successfully created web crawl data source

idstring required

Web crawl data source ID

ingestion_setting_idstring

Web crawl ingestion setting ID

namestring required

Name of the web crawl ingestion setting

start_urlstring required

Start URL of the web crawl

creation_timeinteger

Creation time (Unix timestamp in seconds)

status'unknown' | 'queued' | 'not_started' | 'managed' | 'starting' | 'started' | 'success' | 'failure' | 'canceling' | 'canceled' | 'deleting' | 'delete_job_failure' required

Job status

start_timeinteger

Job start time (Unix timestamp in seconds)

end_timeinteger

Job end time (Unix timestamp in seconds)

Changes

Changed in 2 of the 71 revisions of this API.3

  • 8bc99b31eabf2See the full diff
    • ○

      added the new optional request property

      new-optional-request-property

    • ○

      added the optional property to the response with the status

      response-optional-property-added

  • 7a3a2c03d6781See the full diff
    • ○

      endpoint added

      endpoint-added

Of the 71 revisions, 18 have no diff computed.