Submit a bounded allowlisted site crawl

Requires the Business plan or higher. A Pro or Hobby key receives 403 with code PLAN_FEATURE_UNAVAILABLE and an upgrade_url. Business and Enterprise keys can submit normally; when the deployment-wide product emergency pause is active, they receive 403 FEATURE_NOT_ENABLED without an upgrade_url and GET /credits reports capabilities.routes["/crawl"].reason as platform. This route is not billed, but the gates still apply: "free" means no credit is charged, not that every account may call it.

Crawls use one queued child job per page. Only HTTPS URLs passing the service SSRF guard and the caller's allowedOrigins policy are eligible. Every top-level page navigation and redirect also enforces HTTPS, allowedOrigins and robots.txt. Discovery includes bounded links from site navigation, headers and footers. Every started child capture uses the normal capture waterfall: included monthly quota, then prepaid credits, then permitted plan overage. There is no separate crawl-page quota. Once all created child items are terminal, the parent is terminalized promptly rather than waiting for a periodic reconciler.

One parent workflow at a time. A single API key may have only one batch or crawl active (pending, processing, retrying, or cancel_requested). A second submission while one is running is refused with 409 and code WORKFLOW_ALREADY_ACTIVE, and the response's existingJobId names the blocking job. Recover by polling that job until it reaches a terminal state, or by cancelling it with DELETE. The limit is per API key, so separate keys on the same account run independently. Note that a submission which times out at the gateway may still be running: the work continues server-side, so a 504 on submit can be followed by this 409 on the next attempt even though nothing looks active from the caller's side.