---
title: "Crawl"
method: POST
path: "/crawl"
---

# Crawl

`POST /crawl`

## Request

- Base URL: `https://api.datafuel.dev`
- URL: `https://api.datafuel.dev/crawl`
- Auth: HTTP bearer

## Request body

- CrawlRequest
  - `url` string, required
  - `ai_prompt` string, nullable
  - `json_schema` object, nullable: Optional schema definition for structured data extraction. Format should follow OpenAI's function calling schema format (https://platform.openai.com/docs/guides/structured-outputs). Example types: - string: "type": "string" - integer: "type": "integer" - number: "type": "number" - boolean: "type": "boolean" - array: "type": "array", "items": {"type": "string"} - object: "type": "object", "properties": {...}
  - `javascript_scenario` object[], nullable
  - `depth` integer: The depth of the crawl 1 depth mean only the first level of links will be scraped like https://example.com/page1 and https://example.com/page2
  - `limit` integer: The maximum number of pages to scrape
  - `exclusion_pattern` string: Regex pattern to exclude specific URLs (e.g., 'https://.*datafuel\.dev/blog/.*' to exclude blog pages)
  - `excluded_links` string: Comma-separated list of URLs to exclude from crawling

## Response `200`

Successful Response

- ScrapeResponse
  - `job_id` string, required: The identifier for the scraping job

## Other responses

- `400`: Bad Request
- `401`: Unauthorized
- `422`: Validation Error

---

[API](https://skmtc.dev/datafuel/apis/datafuel-api.md) - [All operations](https://skmtc.dev/datafuel/apis/datafuel-api/llms.txt) - [OpenAPI document](https://skmtc.dev/datafuel/apis/datafuel-api/revisions/aff86ad8da21?raw)