XOxorbitsai
kb

Ingest Web

Changed on

Ingest website content into the knowledge base.

Args: collection: Target collection name start_url: Starting URL for crawling max_pages: Maximum number of pages to crawl max_depth: Maximum crawl depth url_patterns: Comma-separated URL match patterns (regex) exclude_patterns: Comma-separated exclusion patterns (regex) same_domain_only: Only crawl same domain content_selector: CSS selector for main content area remove_selectors: Comma-separated CSS selectors to remove concurrent_requests: Number of concurrent requests request_delay: Delay between requests in seconds timeout: Request timeout in seconds respect_robots_txt: Respect robots.txt rules parse_method: Parser for document ingestion chunk_strategy: Chunking strategy chunk_size: Chunk size in characters chunk_overlap: Chunk overlap in characters embedding_model_id: Embedding model ID embedding_batch_size: Batch size for embedding max_retries: Maximum retry attempts retry_delay: Delay between retries

post/api/kb/ingest-web

Request

Request body application/x-www-form-urlencoded

collectionstring required

Target collection name

start_urlstring required

Starting URL for crawling

max_pagesinteger nullable

Maximum number of pages to crawl (default: 100)

max_depthinteger nullable

Maximum crawl depth (default: 3)

url_patternsstring nullable

Comma-separated URL match patterns (regex)

exclude_patternsstring nullable

Comma-separated exclusion patterns (regex)

same_domain_onlyboolean nullable

Only crawl same domain (default: True)

content_selectorstring nullable

CSS selector for main content area

remove_selectorsstring nullable

Comma-separated CSS selectors to remove

concurrent_requestsinteger nullable

Concurrent requests (default: 3, max: 10)

request_delaynumber nullable

Delay between requests in seconds (default: 1.0)

timeoutinteger nullable

Request timeout in seconds (default: 30)

respect_robots_txtboolean nullable

Respect robots.txt (default: True)

parse_method'default' | 'pypdf' | 'pdfplumber' | 'unstructured' | 'pymupdf' | 'deepdoc'

Available parsing methods

chunk_strategy'recursive' | 'fixed_size' | 'markdown'

Available chunk strategies

chunk_sizeinteger nullable

Chunk size in characters (default: 1000)

chunk_overlapinteger nullable

Chunk overlap (default: 200)

separatorsstring nullable

Custom chunk separators as JSON array of strings; only used when chunk_strategy is recursive.

embedding_model_idstring

Embedding model ID

embedding_batch_sizeinteger nullable

Batch size for embedding (default: 10)

max_retriesinteger nullable

Maximum retries for embedding failures (default: 3)

retry_delaynumber nullable

Delay between retries in seconds (default: 1.0)

Response

Successful Response

statusstring required

Overall status: success|error|partial

collectionstring required

Target collection name

total_urls_foundinteger required

Total number of unique URLs discovered

pages_crawledinteger required

Number of successfully crawled pages

pages_failedinteger required

Number of pages that failed to crawl

documents_createdinteger required

Number of documents created in collection

chunks_createdinteger required

Total number of chunks created

embeddings_createdinteger required

Total number of embeddings generated

crawled_urlsstring[]

List of successfully crawled URLs

failed_urlsobject

Map of failed URLs to error messages

messagestring required

Human-readable summary message

warningsstring[]

Non-critical warnings

side_effects_may_remainboolean

Whether rollback failed after an ingestion error and persisted side effects may still exist

elapsed_time_msinteger required

Total elapsed time in milliseconds

Changes