---
title: "Set up website-crawl knowledge base"
method: PUT
path: "/v1/agents/{id}/knowledge-bases/website"
---

# Set up website-crawl knowledge base

`PUT /v1/agents/{id}/knowledge-bases/website`

## Request

- Base URL: `https://api.runbear.io`
- URL: `https://api.runbear.io/v1/agents/{id}/knowledge-bases/website`
- Auth: HTTP bearer

Configure the website-crawl knowledge base for an agent. PUT semantics: the request fully replaces the set of crawl roots. For each rootUrl already present in the agent's configuration, the existing dashboard-set per-page selections (nodeSelections) are preserved and only excludedUrls are overwritten. A rootUrl with no per-page selections is seeded with its root page, which the crawl starts from and expands to discover linked pages beneath it. Existing rootUrls absent from the request are removed. Pass an empty configs array to stop crawling entirely. A website sync is enqueued on every successful PUT.

## Path parameters

- `id` string, uuid, required: The ID of the agent to retrieve

## Request body

- WebsiteKBUpdateRequest: Website-crawl knowledge base update request
  - `configs` WebsiteKBConfigInput[], required: Full set of website crawl configurations. PUT semantics: existing rootUrls in this list have their excludedUrls overwritten and their dashboard-set per-page selections preserved; a rootUrl with no per-page selections is seeded with its root page, which the crawl starts from; existing rootUrls absent from this list are removed. Pass an empty array to stop crawling entirely. Each rootUrl must appear at most once.
    - `rootUrl` string, uri, required: Root URL to crawl. Must be a fully-qualified URL (e.g., https://example.com). It is canonicalized before storage (hash dropped, trailing slash ensured on directory-like paths and stripped from file-extension paths), so https://example.com and https://example.com/ refer to the same root.
    - `excludedUrls` string[]: Fully-qualified URLs to exclude from the crawl under this root URL. Each must be an absolute URL (e.g., https://example.com/admin); everything at or below it is skipped. Relative paths or glob patterns are rejected.

## Response `200`

Default Response

- WebsiteKnowledgeBase: Runbear-managed website crawl knowledge base for the agent
  - `configs` WebsiteKnowledgeBaseConfig[], required: Crawl configurations for the website knowledge base
    - `rootUrl` string, required: Root URL crawled for this knowledge base entry
    - `excludedUrls` string[]: URLs excluded from the crawl
    - `nodeSelections` KnowledgeBaseNodeSelection[], required: Specific pages/subtrees selected under the root URL
      - `id` string, required: Selection node ID. For Google Drive: the Drive/folder/file ID. For Website: the URL.
      - `ancestorIds` string[], required: Ancestor node IDs, root first
      - `excludedChildIds` string[], required: Node IDs explicitly excluded from sync
      - `disableSyncNewChildren` boolean: When true, newly discovered children of this node are not auto-synced
  - `syncStatus` KnowledgeBaseSyncStatus, required: Sync status snapshot for a knowledge base. lastHeartbeatAt and internal error metadata are omitted.
    - `state` 'IN_PROGRESS' | 'SUCCESS' | 'PARTIAL_SUCCESS' | 'FAILED' | 'SUSPENDED', required: Current sync state for this knowledge base
    - `lastSuccessAt` string, nullable, required: Timestamp of the last fully successful sync, or null if never completed
    - `lastDurationMs` integer, nullable, required: Duration of the last sync attempt in milliseconds
    - `error` object, nullable, required: Most recent sync error, if any
      - `type` string, required: Error category (server_error, authentication, authorization, disintegrated, …)
      - `message` string, required: Human-readable error message

## Other responses

- `400`: Default Response
- `404`: Default Response

---

[API](https://skmtc.dev/runbear/apis/runbear-api.md) - [All operations](https://skmtc.dev/runbear/apis/runbear-api/llms.txt) - [OpenAPI document](https://skmtc.dev/runbear/apis/runbear-api/revisions/c6a9c3066a5a?raw)