DEdevflowinc
Crawl

Create a new crawl request

Changed on

This endpoint is used to create a new crawl request for a dataset. The request payload should contain the crawl options to use for the crawl.

post/api/crawl

Request

  • Base URL
    https://api.trieve.ai
  • URL
    https://api.trieve.ai/api/crawl
  • Auth
    API key in header Authorization

Headers

TR-Datasetstring uuid required

The dataset id to use for the request

Request body

Example request

{
  "crawl_options": {
    "crawl_options": {
      "allow_external_links": false,
      "boost_titles": true,
      "exclude_tags": [
        "#ad",
        "#footer",
        "header",
        "head",
        "navbar",
        "footer",
        "aside",
        "nav",
        "form"
      ],
      "heading_remove_strings": [
        "Advertisement",
        "Sponsored"
      ],
      "ignore_sitemap": true,
      "include_tags": [],
      "interval": "daily",
      "limit": 50,
      "site_url": "nedzo.ai"
    }
  }
}

Response

Crawl created successfully

attempt_numberinteger required
crawl_type'firecrawl' | 'openapi' | 'shopify' | 'youtube' required
created_atstring date-time required
dataset_idstring uuid required
idstring uuid required
intervalstring nullable
next_crawl_atstring date-time nullable
scrape_idstring uuid required
urlstring required

Example response

{
  "crawl_options": {
    "crawl_options": {
      "allow_external_links": false,
      "boost_titles": true,
      "exclude_tags": [
        "#ad",
        "#footer",
        "header",
        "head",
        "navbar",
        "footer",
        "aside",
        "nav",
        "form"
      ],
      "heading_remove_strings": [
        "Advertisement",
        "Sponsored"
      ],
      "ignore_sitemap": true,
      "include_tags": [],
      "interval": "daily",
      "limit": 50,
      "site_url": "nedzo.ai"
    }
  }
}

Changes

  • a77fb2f99507Revision changes
    • info

      added the new optional request property /

    • info

      added the optional property / to the response with the status

  • 661f8d6f359bRevision changes
    • warning

      removed the request property /

    • warning

      removed the optional property / from the response with the status