Crawl & batch

Get crawl status

Poll a crawl for its progress and per-page results.

get/v1/web/crawl/{id}

Request

Base URL: https://api.hydrafetch.com
URL: https://api.hydrafetch.com/v1/web/crawl/{id}
Auth: API key in header X-API-Key

Path parameters

idstring required

The crawl id returned when the crawl started.

Response

Example response

{
  "data": {
    "id": "019f3c09-6fae-740f-9257-10c2b6af7f43",
    "kind": "crawl",
    "status": "running",
    "total": 100,
    "completed": 42,
    "failed": 1,
    "creditsUsed": 42,
    "pages": [
      {
        "url": "https://example.com/blog/post",
        "requestedUrl": "https://example.com/blog/post/",
        "status": "completed",
        "depth": 1,
        "discoveredVia": "sitemap",
        "data": {
          "url": "https://example.com",
          "finalUrl": "https://example.com/",
          "status": 200,
          "metadata": {
            "structure": "markdown",
            "pageType": "article",
            "wordCount": 214
          },
          "usage": {
            "creditsUsed": 1,
            "creditsRemaining": 4999,
            "freshness": "fresh"
          },
          "quality": {
            "confidence": 0.94,
            "complete": true
          },
          "structured": {
            "entities": [
              {
                "type": "Product"
              }
            ]
          }
        }
      }
    ]
  }
}

Changes

No recorded changes to this operation across all 1 revision of this API.