Pages

Site Crawl

Crawl a site and get clean markdown for every page, in one call. Same-host BFS with depth and page-budget limits, robots.txt honoured by default. $0.0015 per requested page (1-25). Price: $0.0015 per requested page (1-25); an x402 payment is at least $0.004.

post/crawl

Request

  • Base URL: https://api.weblens.dev
  • URL: https://api.weblens.dev/crawl
  • Auth: API key in header Payment-Signature

Request body

urlstring uri required

The URL to process

limitnumber

Page budget — you are charged per requested page

maxDepthnumber

Link depth from the start URL (0 = start page only)

includestring[]

Only crawl URLs whose path+query contains one of these

excludestring[]

Skip URLs whose path+query contains one of these

respectRobotsboolean

Honour robots.txt (default true; disable only for sites you control)

maxCharsnumber

Per-page content character cap

timeoutnumber

Request timeout in milliseconds

Example request

{
  "url": "https://example.com",
  "limit": 10,
  "maxDepth": 2
}

Response

Whole-site crawl returning markdown per page, in one call.

urlstring
crawledAtstring
requestIdstring

Changes

No recorded changes to this operation across all 1 revision of this API.