crawls

Create a web crawl data source

Changed on

Creates a new web crawl data source and starts ingestion.

Required scope: `ingestion:manage`

post/crawls

Request

  • Base URL
    https://developer.qaip.com/api/v1
  • URL
    https://developer.qaip.com/api/v1/crawls
  • Auth
    API key in header x-api-key

Request body

namestring required

Name of the web crawl data source

start_urlstring required

Start URL of the web crawl

path_filtersstring[]

Path filters for crawling. The total number of characters across all elements in the array must be 2000 or fewer.

content_patternstring[]

Content patterns for filtering. The total number of characters across all elements in the array must be 2000 or fewer.

max_depthinteger required

Maximum crawl depth

max_num_filesinteger required

Maximum number of files to crawl

html_onlyboolean

When true, only HTML files will be downloaded

use_browserboolean

Whether to use a headless browser for crawling

no_canonical_checkboolean

Whether to disable duplicate filtering by canonical URL

file_extensionsstring[]

File extensions to include (e.g. ".pdf", ".docx"). For supported file extensions, please refer to https://developer.qaip.com/docs/datasources#%E5%AF%BE%E5%BF%9C%E3%81%97%E3%81%A6%E3%81%84%E3%82%8B%E3%83%95%E3%82%A1%E3%82%A4%E3%83%AB%E5%BD%A2%E5%BC%8F

rrulestring

Recurrence rule (RFC 5545 RRULE)

Response

Successfully created web crawl data source

idstring required

Web crawl data source ID

ingestion_setting_idstring

Web crawl ingestion setting ID

namestring required

Name of the web crawl ingestion setting

start_urlstring required

Start URL of the web crawl

creation_timeinteger

Creation time (Unix timestamp in seconds)

status'unknown' | 'queued' | 'not_started' | 'managed' | 'starting' | 'started' | 'success' | 'failure' | 'canceling' | 'canceled' | 'deleting' | 'delete_job_failure' required

Job status

start_timeinteger

Job start time (Unix timestamp in seconds)

end_timeinteger

Job end time (Unix timestamp in seconds)

Changes

  • 2489c6f1a9d8Revision changes
    • info

      added the non-success response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

  • b065e544fec5Revision changes
    • info

      added the new optional request property