---
title: "Crawl a public website into a knowledge base"
method: POST
path: "/v1/knowledgebases/{id}/crawls"
tags: ["Knowledge Bases"]
---

# Crawl a public website into a knowledge base

`POST /v1/knowledgebases/{id}/crawls`

## Request

- Base URL: `https://api.pyai.com`
- URL: `https://api.pyai.com/v1/knowledgebases/{id}/crawls`
- Auth: one of:
  - HTTP bearer
  - API key in header `x-api-key`

Discover same-origin pages from a public seed URL (robots.txt / sitemap, then a shallow link walk), rank them, and register each selected page as a normal URL document. Private, loopback, and metadata addresses are rejected. Defaults to 25 pages, hard-capped at 40. Already-ingested URLs in this knowledge base are reused. Poll document `status` until `indexed` before treating the site as ready. Requires the `kb:manage` scope.

## Path parameters

- `id` string, required

## Request body

- object
  - `url` string, uri, required: Public http(s) seed URL.
  - `max_pages` integer: Page cap after ranking. Defaults to 25, never above 40.

## Response `202`

Pages registered (pending ingestion)

- KnowledgebaseCrawl
  - `object` string
  - `seed_url` string, uri
  - `max_pages` integer
  - `discovered` integer: Same-origin candidates considered before ranking.
  - `accepted` integer: New URL documents registered by this crawl.
  - `reused` integer: Selected URLs that already existed in this knowledge base.
  - `skipped` object[]
    - `url` string
    - `reason` string
  - `documents` object[]
    - `id` string
    - `url` string, uri
    - `title` string
    - `reused` boolean

## Other responses

- `400`: Invalid request (bad field, unsupported value)
- `404`: Resource not found (`code: not_found`)

---

[API](https://skmtc.dev/pyai/apis/pyai-api.md) · [All operations](https://skmtc.dev/pyai/apis/pyai-api/llms.txt) · [OpenAPI document](https://skmtc.dev/pyai/apis/pyai-api/revisions/271a1539185b?raw)