Sign in
crawl-url-lists

Create a web crawl data source from a URL list

Creates a new web crawl data source by directly downloading a list of URLs. Unlike the standard crawl endpoint, this does not follow links or crawl pages — it downloads the specified URLs directly.

Required scope: `ingestion:manage`

post/crawl-url-lists

Request body

namestring required

Name of the web crawl data source

target_urlsstring[] required

List of URLs to download directly. Each URL must use http or https scheme.

max_num_filesinteger

Maximum number of files to download. Defaults to the number of target URLs if omitted.

no_canonical_checkboolean

Whether to disable duplicate filtering by canonical URL

rrulestring

Recurrence rule (RFC 5545 RRULE)

Response

Successfully created web crawl data source from URL list

idstring required

Web crawl data source ID

ingestion_setting_idstring

Web crawl ingestion setting ID

namestring required

Name of the web crawl ingestion setting

start_urlstring required

Start URL of the web crawl

creation_timeinteger

Creation time (Unix timestamp in seconds)

status'unknown' | 'queued' | 'not_started' | 'managed' | 'starting' | 'started' | 'success' | 'failure' | 'canceling' | 'canceled' | 'deleting' | 'delete_job_failure' required

Job status

start_timeinteger

Job start time (Unix timestamp in seconds)

end_timeinteger

Job end time (Unix timestamp in seconds)

Changes

Changed in 1 of the 71 revisions of this API.1

Of the 71 revisions, 18 have no diff computed.