---
title: "Site Crawl"
method: POST
path: "/crawl"
tags: ["Pages"]
---

# Site Crawl

`POST /crawl`

## Request

- Base URL: `https://api.weblens.dev`
- URL: `https://api.weblens.dev/crawl`
- Auth: API key in header `Payment-Signature`

Crawl a site and get clean markdown for every page, in one call. Same-host BFS with depth and page-budget limits, robots.txt honoured by default. $0.0015 per requested page (1-25). Price: $0.0015 per requested page (1-25); an x402 payment is at least $0.004.

## Request body

- object
  - `url` string, uri, required: The URL to process
  - `limit` number: Page budget — you are charged per requested page
  - `maxDepth` number: Link depth from the start URL (0 = start page only)
  - `include` string[]: Only crawl URLs whose path+query contains one of these
  - `exclude` string[]: Skip URLs whose path+query contains one of these
  - `respectRobots` boolean: Honour robots.txt (default true; disable only for sites you control)
  - `maxChars` number: Per-page content character cap
  - `timeout` number: Request timeout in milliseconds

## Response `200`

Whole-site crawl returning markdown per page, in one call.

- object
  - `url` string
  - `pages` object[]: Per-page results with url, depth, status, title, content, truncated, error
    - `url` string
    - `depth` integer
    - `status` 'success' | 'failed'
    - `title` string
    - `content` string: Page as clean markdown
    - `truncated` boolean
    - `error` string
  - `summary` object: crawled, successful, failed, discovered, limit, maxDepth, robotsRespected
    - `crawled` integer
    - `successful` integer
    - `failed` integer
    - `discovered` integer
    - `limit` integer
    - `maxDepth` integer
    - `robotsRespected` boolean
  - `crawledAt` string
  - `requestId` string

## Other responses

- `400`: The body failed validation
- `402`: Payment required — x402 v2 protocol. Parse the PAYMENT-REQUIRED response header (base64-encoded JSON), sign the payment with your wallet, then retry the request with the Payment-Signature request header.
- `403`: robots.txt disallows crawling the start URL. Set respectRobots=false only for sites you control.

---

[API](https://skmtc.dev/weblens/apis/weblens-api.md) · [All operations](https://skmtc.dev/weblens/apis/weblens-api/llms.txt) · [OpenAPI document](https://skmtc.dev/weblens/apis/weblens-api/revisions/63a7e81b6921?raw)