---
title: "Submit a bounded allowlisted site crawl"
method: POST
path: "/crawl"
tags: ["Crawl"]
---

# Submit a bounded allowlisted site crawl

`POST /crawl`

## Request

- Base URL: `https://api.domscout.io`
- URL: `https://api.domscout.io/crawl`
- Auth: API key in header `x-api-key`

Requires the Business plan or higher. A Pro or Hobby key receives 403 with code PLAN_FEATURE_UNAVAILABLE and an upgrade_url. Business and Enterprise keys can submit normally; when the deployment-wide product emergency pause is active, they receive 403 FEATURE_NOT_ENABLED without an upgrade_url and GET /credits reports capabilities.routes["/crawl"].reason as platform. This route is not billed, but the gates still apply: "free" means no credit is charged, not that every account may call it.

Crawls use one queued child job per page. Only HTTPS URLs passing the service SSRF guard and the caller's allowedOrigins policy are eligible. Every top-level page navigation and redirect also enforces HTTPS, allowedOrigins and robots.txt. Discovery includes bounded links from site navigation, headers and footers. Every started child capture uses the normal capture waterfall: included monthly quota, then prepaid credits, then permitted plan overage. There is no separate crawl-page quota. Once all created child items are terminal, the parent is terminalized promptly rather than waiting for a periodic reconciler.

**One parent workflow at a time.** A single API key may have only one batch or crawl active (pending, processing, retrying, or cancel_requested). A second submission while one is running is refused with 409 and code WORKFLOW_ALREADY_ACTIVE, and the response's existingJobId names the blocking job. Recover by polling that job until it reaches a terminal state, or by cancelling it with DELETE. The limit is per API key, so separate keys on the same account run independently. Note that a submission which times out at the gateway may still be running: the work continues server-side, so a 504 on submit can be followed by this 409 on the next attempt even though nothing looks active from the caller's side.

## Headers

- `Idempotency-Key` string

## Request body

- CrawlRequest
  - `seedUrl` string, uri, required: HTTPS starting URL.
  - `allowedOrigins` string[], required: Explicit HTTPS origin allowlist, enforced on every top-level page navigation and redirect together with robots.txt. Include every top-level page origin the crawler may follow.
  - `includePatterns` string[]
  - `excludePatterns` string[]
  - `maxDepth` integer
  - `maxPages` integer
  - `respectRobots` true: robots.txt is always respected for crawls; false is not accepted.
  - `capture` CrawlCaptureOptions: Per-page screenshot options. url, html, async, callbackUrl, callbackSecret, and save are owned by the crawl job and are not accepted here.
    - `format` 'png' | 'jpeg' | 'webp' | 'pdf'
    - `fullPage` boolean
    - `extractMarkdown` boolean
    - `semanticSnapshot` union
      - boolean: true enables the default rich snapshot.
      - object
        - `enabled` boolean
        - `maxNodes` integer
        - `detail` 'summary' | 'full'
        - `selectorBundles` boolean
        - `timing` 'beforeActions' | 'afterActions' | 'both'
        - `includeIframes` boolean
    - `diagnostics` union
      - boolean: true enables all bounded, redacted diagnostics.
      - object
        - `console` boolean
        - `pageErrors` boolean
        - `network` boolean
        - `resources` boolean
        - `timing` boolean
        - `maxEntries` integer
    - `render` RenderOptions
      - `disableAnimations` boolean
      - `colorScheme` 'light' | 'dark' | 'no-preference'
      - `locale` string: BCP 47 locale such as en-US.
      - `timezone` string: IANA timezone such as Asia/Kolkata.
      - `geolocation` Geolocation
        - `latitude` number, required
        - `longitude` number, required
        - `accuracy` number
      - `userAgent` string: Overrides the browser User-Agent string. The API enforces 1024 characters; a longer value is rejected with 400.
      - `transparentBackground` boolean
      - `blockResourceTypes` string[]
      - `blockDomains` string[]
      - `capturePadding` integer
      - `devicePreset` 'desktop' | 'iphone13' | 'pixel7' | 'ipad': Named viewport and device-emulation preset.
      - `reducedMotion` boolean
      - `media` 'screen' | 'print'
    - `cleanup` union
      - boolean: true enables all built-in heuristic cleanup categories.
      - object
        - `enabled` boolean
        - `preserveSelectors` string[]: Elements matching these selectors are never hidden by heuristic cleanup.
    - `extract` ExtractOptions
      - `fields` object, required
      - `strict` boolean: Fail the request when a required field is missing.
  - `metadata` object
  - `callbackUrl` string, uri: Public HTTPS callback endpoint checked against private, reserved, and metadata addresses before acceptance and delivery.

## Response `200`

Idempotent replay of a crawl workflow that has already reached a terminal state (done, error, or cancelled). The body carries replayed:true and the final counts. A replay of a workflow that is still running returns 202 instead.

- ParentJobAccepted
  - `status` 'accepted' | 'pending' | 'processing' | 'retrying' | 'done' | 'error' | 'cancel_requested' | 'cancelled', required
  - `jobId` string, uuid, required
  - `pollUrl` string, required
  - `kind` 'batch' | 'crawl', required
  - `counts` ChildCounts
    - `total` integer
    - `pending` integer
    - `processing` integer
    - `done` integer
    - `completed` integer
    - `succeeded` integer
    - `error` integer
    - `failed` integer
    - `cancelled` integer
  - `createdAt` string, date-time
  - `message` string
  - `replayed` boolean: Present and true when an Idempotency-Key matched an existing workflow and this response replays it rather than creating a new one.

## Other responses

- `202`: Crawl parent job accepted.
- `400`: The request failed validation.
- `403`: The request was refused. Returned when the x-api-key header is missing, revoked, or invalid, and when a valid key is blocked by a plan entitlement, the trusted-JavaScript account entitlement, a global product pause, or a security policy. A plan-only restriction returns code PLAN_FEATURE_UNAVAILABLE and can include upgrade_url. FEATURE_NOT_ENABLED identifies either the global product emergency pause or an eligible Business account without trusted-JavaScript permission; it carries no upgrade_url, so upgrading or retrying cannot clear it. Browser capture can return BROWSER_ISOLATION_REQUIRED or BROWSER_EXECUTION_DENIED with reason public_capture_not_enabled when browser work is unavailable for your account. These platform restrictions carry no upgrade_url and are not resolved by changing plan or key. Check GET /credits for current account capabilities. The API does not use 401: it authenticates with an API key header rather than an HTTP authentication scheme, so there is no WWW-Authenticate challenge to issue.
- `409`: The Idempotency-Key was already used with a different request body.
- `429`: The plan quota or rate limit has been reached.
- `503`: A dependency this request needs is unavailable or not configured for this deployment. Codes: SUPABASE_TIMEOUT and SUPABASE_INVALID_RESPONSE (a storage operation did not answer, or answered unreadably — the write may have committed, so any retry must be idempotent), CAPTURE_ARTIFACT_UNAVAILABLE and CAPTURE_ARTIFACT_CORRUPT (a stored artifact could not be read or parsed), ASYNC_JOBS_NOT_CONFIGURED and CAPTURE_RETRIEVAL_NOT_CONFIGURED (the capability is not configured here, on any plan). Retryable unless the code names a deployment configuration.

---

[API](https://skmtc.dev/domscout/apis/domscout-api.md) · [All operations](https://skmtc.dev/domscout/apis/domscout-api/llms.txt) · [OpenAPI document](https://skmtc.dev/domscout/apis/domscout-api/revisions/20562d5b5d94?raw)