From url and (unless sitemap: skip) the site's sitemaps, same host (optionally subdomains), up to limit pages and max_depth link hops, filtered by include_paths / exclude_paths (regular expressions matched against the URL path). Breadth first, or best first with query (pages whose URL and link text match it are read first). Each host's robots.txt is honoured (RFC 9309: rules, Crawl-delay; a robots.txt that answers 5xx stops the crawl at the start page). URLs are canonicalised (fragments and tracking parameters dropped, www., trailing slashes and redirects folded) so each page is read once. Each page comes back as Markdown (or text) with its metadata. A crawl runs for at most 30 minutes and returns up to 16 MB of pages; result.stats.stopped says why it ended.

Returns 202 with a task; poll GET /v1/crawl/{id} or wait for the signed crawl.completed webhook. Pages read cost 1 credit each (0.5 from cache); a crawl that read nothing fails (no_pages) and is refunded. At most 20 crawls per account may be queued or running at once (429); a crawl interrupted by a deploy restarts from the beginning.