DIdiffbot

Create a Crawl

Create and start a job to spider and extract pages through a site.

post/crawl

Request

Request body application/x-www-form-urlencoded

namestring required

Job name. This should be a unique identifier and can be used to modify your crawl or retrieve its output.

seedsstring required

Seed URL(s). Must be URL encoded. Separate multiple URLs with whitespace to spider multiple sites within the same crawl. If the seed contains a non-www subdomain ("https://blog.diffbot.com" or "https://docs.diffbot.com") Crawl will restrict spidering to the specified subdomain.

apiUrlstring required

Full Extract API URL through which to process pages. E.g., &apiUrl=https://api.diffbot.com/v3/analyze to process matching links via the Analyze API, which will automatically determine the page type to extract as. The Extract API URL can include querystring parameters to tailor the output. <p>For example, &apiUrl=https://api.diffbot.com/v3/product?fields=querystring,meta will process matching links using the Product API, and also return the querystring and meta fields.

Response

200

responsestring

Example response

{
  "response": "Successfully added urls for spidering.",
  "jobs": [
    {
      "maxToProcessPerSubdomain": -1,
      "jobStatus": {
        "message": "Job is initializing."
      },
      "maxHops": -1,
      "onlyProcessIfNew": 1,
      "downloadJson": "https://api.diffbot.com/v3/crawl/download/<REDACTED>.json",
      "maxRounds": -1,
      "restrictDomain": 1,
      "type": "crawl",
      "apiUrl": "https://api.diffbot.com/v3/analyze",
      "useCanonical": 1,
      "maxToCrawlPerSubdomain": -1,
      "jobCreationTimeUTC": 1726163287,
      "downloadUrls": "https://api.diffbot.com/v3/crawl/download/<REDACTED>.csv",
      "obeyRobots": 1,
      "maxToProcess": 100000,
      "seedRecrawlFrequency": -1,
      "seeds": "https://example.com",
      "crawlDelay": 0.25,
      "currentTime": 1726163288,
      "maxToCrawl": 100000,
      "currentTimeUTC": 1726163288,
      "name": "test-crawl"
    }
  ]
}

Changes

No recorded changes to this operation across all 1 revision of this API.