---
title: "Crawl Domain"
method: POST
path: "/chatbots/{chatbot_id}/websites/crawl"
tags: ["Chatbot Websites"]
---

# Crawl Domain

`POST /chatbots/{chatbot_id}/websites/crawl`

## Request

- Base URL: `https://app.boei.help/api/v1`
- URL: `https://app.boei.help/api/v1/chatbots/{chatbot_id}/websites/crawl`
- Auth: HTTP bearer

Start crawling a domain to discover and train pages. Pages are automatically discovered, crawled, and processed.

NOTE: Crawling runs in the background and can take minutes for large sites. Poll websites.get for page counts per status. Do not start the same crawl twice.

## Request body

- object
  - `url` string, required: The domain URL to crawl.
  - `name` string, nullable: Optional display name. Auto-generated from URL if omitted.
  - `include_paths` string, nullable: Comma-separated paths to include.
  - `exclude_paths` string, nullable: Comma-separated paths to exclude.

## Response `202`

- object
  - `website` object
    - `id` string
    - `name` string
    - `url` string
    - `type` string
    - `status` string
    - `progress` integer
  - `job_id` string
  - `status_url` string
  - `message` string

---

[API](https://skmtc.dev/boei/apis/boei-api.md) - [All operations](https://skmtc.dev/boei/apis/boei-api/llms.txt) - [OpenAPI document](https://skmtc.dev/boei/apis/boei-api/revisions/fc439667d2d4?raw)