BOboei
Chatbot Websites

Crawl Domain

Start crawling a domain to discover and train pages. Pages are automatically discovered, crawled, and processed.

NOTE: Crawling runs in the background and can take minutes for large sites. Poll websites.get for page counts per status. Do not start the same crawl twice.

post/chatbots/{chatbot_id}/websites/crawl

Request

  • Base URL
    https://app.boei.help/api/v1
  • Auth
    HTTP bearer

Request body

urlstring required

The domain URL to crawl.

namestring nullable

Optional display name. Auto-generated from URL if omitted.

include_pathsstring nullable

Comma-separated paths to include.

exclude_pathsstring nullable

Comma-separated paths to exclude.

Example request

{
  "url": "https://example.com",
  "name": "My Website",
  "include_paths": "/docs,/blog",
  "exclude_paths": "/admin,/login"
}

Response

job_idstring
status_urlstring
messagestring

Example response

{
  "website": {
    "id": "550e8400-e29b-41d4-a716-446655440000",
    "name": "Crawl example.com",
    "url": "https://example.com",
    "type": "crawler",
    "status": "crawling",
    "progress": 0
  },
  "job_id": "550e8400-e29b-41d4-a716-446655440000",
  "status_url": "https://app.boei.help/api/v1/chatbots/8a832d1b-0f49-47f6-a05f-0b65b91d2107/websites/550e8400-e29b-41d4-a716-446655440000",
  "message": "Domain crawl started"
}

Changes

No recorded changes to this operation across all 1 revision of this API.