DEdevflowinc
Crawl

Get all crawl requests for a dataset

Changed on

This endpoint is used to get all crawl requests for a dataset.

get/api/crawl

Request

  • Base URL
    https://api.trieve.ai
  • URL
    https://api.trieve.ai/api/crawl
  • Auth
    API key in header Authorization

Query parameters

pageinteger nullable

The page number to retrieve

limitinteger nullable

The number of items to retrieve per page

Headers

TR-Datasetstring uuid required

The dataset id to use for the request

Response

Crawl requests retrieved successfully

attempt_numberinteger required
crawl_type'firecrawl' | 'openapi' | 'shopify' | 'youtube' required
created_atstring date-time required
dataset_idstring uuid required
idstring uuid required
intervalstring nullable
next_crawl_atstring date-time nullable
scrape_idstring uuid required
urlstring required

Example response

[
  {
    "crawl_options": {
      "crawl_options": {
        "allow_external_links": false,
        "boost_titles": true,
        "exclude_tags": [
          "#ad",
          "#footer",
          "header",
          "head",
          "navbar",
          "footer",
          "aside",
          "nav",
          "form"
        ],
        "heading_remove_strings": [
          "Advertisement",
          "Sponsored"
        ],
        "ignore_sitemap": true,
        "include_tags": [],
        "interval": "daily",
        "limit": 50,
        "site_url": "nedzo.ai"
      }
    }
  }
]

Changes