Knowledge Bases

Crawl a public website into a knowledge base

Discover same-origin pages from a public seed URL (robots.txt / sitemap, then a shallow link walk), rank them, and register each selected page as a normal URL document. Private, loopback, and metadata addresses are rejected. Defaults to 25 pages, hard-capped at 40. Already-ingested URLs in this knowledge base are reused. Poll document status until indexed before treating the site as ready. Requires the kb:manage scope.

post/v1/knowledgebases/{id}/crawls

Request

  • Base URL: https://api.pyai.com
  • URL: https://api.pyai.com/v1/knowledgebases/{id}/crawls
  • Auth: one of:
    • HTTP bearer
    • API key in header x-api-key

Path parameters

idstring required

Request body

urlstring uri required

Public http(s) seed URL.

max_pagesinteger

Page cap after ranking. Defaults to 25, never above 40.

Response

Pages registered (pending ingestion)

objectstring
seed_urlstring uri
max_pagesinteger
discoveredinteger

Same-origin candidates considered before ranking.

acceptedinteger

New URL documents registered by this crawl.

reusedinteger

Selected URLs that already existed in this knowledge base.

Example response

{
  "object": "knowledgebase.crawl"
}

Changes

No recorded changes to this operation across all 1 revision of this API.