Scrape

Scrape a URL

Fetch a single URL and return it in the formats you ask for. Waits for the result by default, returning a job id if it runs long; set async to always return a job id immediately.

post/v1/web/scrape

Request

  • Base URL: https://api.hydrafetch.com
  • URL: https://api.hydrafetch.com/v1/web/scrape
  • Auth: API key in header X-API-Key

Request body

urlstring required

The URL to scrape. Must be http(s); a bare domain such as example.com is read over https.

maxAgenumber

Serve from cache if a capture of this URL is younger than this many milliseconds. Omit for the default 24h window; 0 always fetches fresh. Capped at 7 days.

cacheOnlyboolean

Only serve from cache. If there is no fresh cached copy, return 404 instead of fetching.

storeInCacheboolean

Persist the capture for later re-extraction. Default true.

preferStructureboolean

Preserve document structure (headings, lists, tables) over prose density — good for marketing and service pages. Default off.

asyncboolean

Return a job id immediately instead of waiting for the result. Poll GET /v1/web/scrape/{id}. Opt in only: a synchronous call never degrades to a job id on its own, so you get a page or an error, not two response shapes.

onlyMainContentboolean

Return only the main content, dropping nav/boilerplate. Default true.

includeTagsstring[]

CSS selectors to keep. When set, only matching elements are considered.

excludeTagsstring[]

CSS selectors to strip before extraction.

removeBase64Imagesboolean

Strip inline base64 images from the output. Default true.

blockAdsboolean

Remove common ad and tracking elements. Default true.

includeLinksboolean

Keep inline links in the markdown. By default links are kept wherever the page text carries them. Set true to keep them on every page, or false for the densest prose.

waitFornumber

Extra milliseconds to let the page settle before capture.

timeoutnumber

Overall time budget for the request, in milliseconds.

headersobject

Extra request headers to send when fetching the page.

method'GET' | 'POST'

HTTP method to fetch the URL with. Use POST for endpoints that only answer a POST, such as JSON and GraphQL APIs. Defaults to GET.

bodystring

Request body, sent verbatim. Only used when method is POST. Set the content type yourself via headers; nothing is assumed about the format.

formatsstring[]

What to return. Omit for Markdown only. structured is the page's own data (no LLM, cheapest); summary and json are LLM-backed and cost more.

Example request

{
  "url": "https://example.com",
  "includeTags": [
    "article",
    "main"
  ],
  "excludeTags": [
    ".ad",
    "#comments"
  ],
  "location": {
    "country": "us",
    "languages": [
      "en-US",
      "en"
    ]
  },
  "body": "{\"limit\":25,\"offset\":0}",
  "formats": [
    "markdown"
  ],
  "jsonOptions": {
    "prompt": "Extract the product name, price, and availability."
  }
}

Response

The scraped page.

Example response

{
  "data": {
    "url": "https://example.com",
    "finalUrl": "https://example.com/",
    "status": 200,
    "metadata": {
      "structure": "markdown",
      "pageType": "article",
      "wordCount": 214
    },
    "usage": {
      "creditsUsed": 1,
      "creditsRemaining": 4999,
      "freshness": "fresh"
    },
    "quality": {
      "confidence": 0.94,
      "complete": true
    },
    "structured": {
      "entities": [
        {
          "type": "Product"
        }
      ]
    }
  }
}

Changes

No recorded changes to this operation across all 1 revision of this API.