crawl-url-lists

Create a web crawl data source from a URL list

Changed on

Creates a new web crawl data source by directly downloading a list of URLs. Unlike the standard crawl endpoint, this does not follow links or crawl pages — it downloads the specified URLs directly.

Required scope: `ingestion:manage`

post/crawl-url-lists

Request

  • Base URL
    https://developer.qaip.com/api/v1
  • URL
    https://developer.qaip.com/api/v1/crawl-url-lists
  • Auth
    API key in header x-api-key

Request body

namestring required

Name of the web crawl data source

target_urlsstring[] required

List of URLs to download directly. Each URL must use http or https scheme.

max_num_filesinteger

Maximum number of files to download. Defaults to the number of target URLs if omitted.

no_canonical_checkboolean

Whether to disable duplicate filtering by canonical URL

rrulestring

Recurrence rule (RFC 5545 RRULE)

Response

Successfully created web crawl data source from URL list

idstring required

Web crawl data source ID

ingestion_setting_idstring

Web crawl ingestion setting ID

namestring required

Name of the web crawl ingestion setting

start_urlstring required

Start URL of the web crawl

creation_timeinteger

Creation time (Unix timestamp in seconds)

status'unknown' | 'queued' | 'not_started' | 'managed' | 'starting' | 'started' | 'success' | 'failure' | 'canceling' | 'canceled' | 'deleting' | 'delete_job_failure' required

Job status

start_timeinteger

Job start time (Unix timestamp in seconds)

end_timeinteger

Job end time (Unix timestamp in seconds)

Changes

  • 2489c6f1a9d8Revision changes
    • info

      added the non-success response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      added the optional property / to the response with the status

  • b065e544fec5Revision changes
    • info

      added the new optional request property