Sign in
Tools

Fetch URL body and headers with JS-rendering + bot-protection handling

Fetches the body and headers of any URL with automatic JavaScript rendering and common bot-protection handling. Tune via optional parameters (bodyNeeded, headersNeeded, cleanHtml, maxTimeoutMs, bodyMaxBytes). Response includes body, headers, cleanStats. The cleanHtml flag strips HTML noise (scripts, styles, comments) while preserving text content — significant token-cost reduction for LLM consumption; per-request reduction reported in cleanStats. Returns headersAvailable: false when advanced behavioral fingerprinting prevents header access. Intended for AI agents and customer integrations that need direct URL content access.

post/v1/tools/fetch-url

Request

  • Base URL: https://api.competlab.com
  • URL: https://api.competlab.com/v1/tools/fetch-url
  • Auth: API key in header CL-API-Key

Request body

urlstring required

Target URL to fetch. Must use http:// or https:// and resolve to a public host.

bodyNeededboolean

Include body and contentType in the response. Defaults to service-controlled value when omitted.

headersNeededboolean

Include headers and headersAvailable in the response. Defaults to service-controlled value when omitted. When the target site uses advanced behavioral fingerprinting, headersAvailable is false and headers is an empty object — present, not missing. Branch on headersAvailable, never on whether headers exists.

cleanHtmlboolean

When true and the response content-type is text/html, strip HTML noise (scripts, styles, comments) while preserving text content. Significant token-cost reduction for LLM consumption — per-request reduction reported in cleanStats. Requires bodyNeeded.

maxTimeoutMsnumber

Caller-side timeout budget in milliseconds. Accepted range 1000–120000. Defaults to service-controlled value when omitted.

bodyMaxBytesnumber

Per-request response body cap in bytes. Accepted range 1024–104857600 (1 KiB – 100 MiB). Oversize responses are rejected pre-buffer. Defaults to service-controlled value when omitted.

Example request

{
  "url": "https://stripe.com",
  "bodyNeeded": true,
  "headersNeeded": true,
  "maxTimeoutMs": 60000,
  "bodyMaxBytes": 5242880
}

Response

Example response

{
  "item": {
    "statusCode": 200,
    "body": "<!DOCTYPE html><html>…</html>",
    "headers": {
      "content-type": "text/html; charset=utf-8",
      "strict-transport-security": "max-age=63072000"
    },
    "headersAvailable": true,
    "contentType": "text/html",
    "cleanStats": {
      "originalSize": 248532,
      "cleanedSize": 41218,
      "reductionPercent": 83.4,
      "originalSizeKB": 242.7,
      "cleanedSizeKB": 40.3,
      "savedKB": 202.4
    },
    "finalUrl": "https://stripe.com/",
    "durationMs": 1842
  }
}

Changes

No recorded changes to this operation across all 1 revision of this API.