tavily

Crawl a website with LLM-driven path selection

Multi-page crawl starting from a root URL. Returns each crawled page with its extracted content (unlike map, which returns only URLs). Use instructions to guide the crawler in natural language. Tavily uses an LLM to follow only the paths matching your intent. Use select_* / exclude_* filters (comma-separated regex patterns) to constrain scope.

get/tavily/crawl

Request

  • Base URL: https://www.socialcrawl.dev/v1
  • URL: https://www.socialcrawl.dev/v1/tavily/crawl
  • Auth: API key in header x-api-key

Query parameters

urlstring required

Root URL to begin crawling.

max_depthinteger

Maximum link depth from the root URL. Defaults to 1.

max_breadthinteger

Maximum number of links followed per level (per page). Defaults to 20.

limitinteger

Total number of pages the crawler will process before stopping. Defaults to 50.

instructionsstring

Natural-language instructions for the crawler (e.g. 'Find all product pages with pricing').

select_pathsstring

Comma-separated regex patterns: only crawl URLs whose path matches.

select_domainsstring

Comma-separated regex patterns: only crawl URLs whose domain matches.

exclude_pathsstring

Comma-separated regex patterns: skip URLs whose path matches.

exclude_domainsstring

Comma-separated regex patterns: skip URLs whose domain matches.

allow_externalboolean

Whether to follow links to external domains. Defaults to true.

extract_depth'basic' | 'advanced'

Per-page extraction strategy. basic is faster; advanced handles harder pages.

format'markdown' | 'text'

Output format for extracted content. markdown (default) or text.

categoriesstring

Comma-separated list of category hints to bias the crawl toward.

Headers

Cache-Controlstring

Send no-cache to bypass the response cache and force a live fetch. Billed at the normal endpoint cost; the fresh result is written back to cache for the next caller. Only the no-cache directive triggers this. See the Response Schema guide for details.

Idempotency-Keystring

Optional UUID that makes the request safely retriable. The key is reserved atomically per account before billing. A concurrent retry receives 409 IDEMPOTENCY_IN_PROGRESS with a Retry-After of 1 second while the original is young, growing to a quarter of the time it has been running (at most 30); a crashed reservation expires after 5 minutes. A replay keeps the cached payload immutable except for billing metadata: credits_used becomes 0, idempotent_replay becomes true, and credits_remaining is refreshed to the current balance. A known current balance appears in both the body and X-Credits-Remaining header; no balance row resolves to 0. On a transient lookup failure, body credits_remaining is null and X-Credits-Remaining is omitted. Completed outcomes expire after 24 hours. If the claim store is unavailable, the request fails closed before billing. GET endpoints do not combine idempotent replay with Server-Sent Events: use Accept: application/json when sending this header. The unsupported key-plus-SSE combination is rejected before billing.

Response

Successful response

successboolean required

Whether the request succeeded

platformstring required

Platform name

endpointstring required

API endpoint path

credits_usedinteger required

Number of credits consumed

credits_remaininginteger nullable required

Current account balance, or null when this response could not establish one. Free cache hits always return null because they do not establish an authoritative post-request balance. An idempotency replay also returns null when its current-balance lookup fails. It is never a fabricated zero, so a zero here means the account really is empty. The authoritative reading is the one on a billed response, which comes straight from the deduction.

request_idstring required

Unique request identifier for support

cachedboolean required

Whether the response was served from cache

idempotent_replayboolean

True only when this response is an idempotency replay

Example response

{
  "success": true,
  "platform": "tavily",
  "endpoint": "/v1/tavily/crawl",
  "credits_used": 1,
  "credits_remaining": 4999,
  "request_id": "req-abc123",
  "idempotent_replay": true,
  "meta": {
    "hint": {
      "code": "search_multi",
      "message": "You searched \"matcha\" on 3 platforms one call at a time; /v1/search/multi returns each platform's own search rows in one call at the same price per platform, and a platform with no rows is not charged.",
      "method": "GET",
      "path": "/v1/search/multi?query=matcha&platforms=instagram,tiktok,youtube"
    }
  }
}

Changes

No recorded changes to this operation across all 1 revision of this API.