---
title: "Crawl a website: fetch a page, follow the links inside the same host and return every page it reached, within a page budget you set. For reading documentation, a product catalogue or a changelog in one paid call instead of many."
method: POST
path: "/v1/tools/site-crawl"
---

# Crawl a website: fetch a page, follow the links inside the same host and return every page it reached, within a page budget you set. For reading documentation, a product catalogue or a changelog in one paid call instead of many.

`POST /v1/tools/site-crawl`

## Request

- Base URL: `https://agents.dexl.io`
- URL: `https://agents.dexl.io/v1/tools/site-crawl`
- Auth: API key in header `X-Payment`

Paid per call via x402. Send without payment to get the exact price in the 402.

## Request body

- object
  - `url` string, required: Absolute http(s) URL to start from
  - `max_pages` integer

## Response `200`

The tool output

## Other responses

- `400`: The input did not match the schema
- `402`: Payment required

---

[API](https://skmtc.dev/dexl/apis/dexl-agents.md) · [All operations](https://skmtc.dev/dexl/apis/dexl-agents/llms.txt) · [OpenAPI document](https://skmtc.dev/dexl/apis/dexl-agents/revisions/e3692765c3f7?raw)