---
title: "Start Crawl"
method: POST
path: "/v1/crawl"
tags: ["crawl"]
---

# Start Crawl

`POST /v1/crawl`

## Request

- Base URL: `https://api.snoopscan.com`
- URL: `https://api.snoopscan.com/v1/crawl`
- Auth: none declared

## Headers

- `authorization` string, nullable

## Request body

- CrawlRequest
  - `url` string, required: The page to start from. An absolute http or https URL.
  - `limit` integer: The most pages to crawl, 1 to 100,000.
  - `maxDepth` integer: How many links deep to follow from the start page. 0 is the start page alone.
  - `maxConcurrency` integer: How many pages to fetch at once, 1 to 50.
  - `includePaths` string[]: Only crawl URLs whose path and query match one of these regexes, e.g. `^/blog/`. Empty means every path.
  - `excludePaths` string[]: Skip URLs whose path and query match any of these regexes. Wins over `includePaths`.
  - `allowExternalLinks` boolean: Follow links onto other domains. Off by default, since with a high `limit` a crawl can reach many unrelated sites.
  - `allowBackwardLinks` boolean: Follow links above the starting path on the same host.
  - `ignoreSitemap` boolean: Find pages by following links only, without reading the site's sitemap.
  - `ignoreQueryParameters` boolean: Treat URLs that differ only in their query string as the same page.
  - `deduplicateSimilarURLs` boolean: Collapse URLs that differ only in tracking parameters or a trailing slash.
  - `delay` integer: Minimum milliseconds between requests to the same host.
  - `respectRobots` boolean: Honor the site's robots.txt for this job.
  - `scrapeOptions` ScrapeOptions: How a page is fetched and what comes back: every /v1/scrape field except `url`.
    - `formats` union[]: What to return: `markdown`, `html`, `rawHtml`, `links`, `media`, `summary`, `screenshot`, or a `json` object carrying your schema. Ask for several in one call.
      - union
        - 'markdown' | 'html' | 'rawHtml' | 'links' | 'media' | 'summary' | 'screenshot' | 'json' | 'network'
        - ScreenshotFormat
          - `type` 'screenshot', required
          - `fullPage` boolean
          - `quality` integer
        - JsonFormat
          - `schema` object, nullable: Your own JSON Schema for the fields you want. Give this or `template`, not both.
          - `template` string, nullable: A named template instead of a schema — `product`, `article`, `jobPosting` and more. GET /v1/templates lists them with their fields.
          - `type` 'json', required
          - `prompt` string, nullable
        - ChangeTrackingFormat
          - `type` 'changeTracking', required
          - `modes` string[]
    - `onlyMainContent` boolean: Return the article and drop navigation, headers, footers and cookie banners. Turn it off to keep the whole page.
    - `includeTags` string[]: CSS selectors to keep even when they sit outside the main content.
    - `excludeTags` string[]: CSS selectors to drop before the page is read.
    - `maxAge` integer: Serve a cached copy if one was taken within this many milliseconds. A cache hit costs nothing. Set `0` to force a fresh fetch.
    - `storeInCache` boolean: Keep this response in the cache so a later request can be served free.
    - `waitFor` integer: Milliseconds to wait after the page loads before reading it, for content that loads late. Renders the page in a browser, and is billed as a browser page.
    - `timeout` integer: Give up on this request after this many milliseconds.
    - `actions` union[]: Steps to run in a browser before the page is read: click, type, scroll, wait or `captchaCheckbox`. A `captchaCheckbox` step needs `successSelector`.
      - union
        - WaitAction
          - `type` 'wait', required
          - `milliseconds` integer, nullable
          - `selector` string, nullable
        - ClickAction
          - `type` 'click', required
          - `selector` string, required
        - CaptchaCheckboxAction: Click a challenge checkbox once, then wait for `successSelector` to appear.
          - `type` 'captchaCheckbox', required
          - `frameSelector` string
          - `selector` string
          - `successSelector` string, required
          - `timeoutMs` integer
          - `captureEvidence` boolean: Return a screenshot from before and after the click.
        - WriteAction
          - `type` 'write', required
          - `selector` string, required
          - `text` string, required
        - PressAction
          - `type` 'press', required
          - `key` string, required
        - ScrollAction
          - `type` 'scroll', required
          - `direction` 'up' | 'down'
          - `amount` integer
        - ScreenshotAction
          - `type` 'screenshot', required
          - `fullPage` boolean
        - ScrapeAction
          - `type` 'scrape', required
        - ExecuteJavascriptAction
          - `type` 'executeJavascript', required
          - `script` string, required
    - `captchaHandling` 'auto' | 'off': Click a challenge checkbox once if one appears. Set `off` to never click.
    - `captchaEvidence` boolean: Return screenshots from before and after an automatic checkbox click.
    - `headers` object: Extra headers to send. `User-Agent` and other browser-identity headers are set for you and cannot be changed.
    - `mobile` boolean: Read the page as a phone would see it.
    - `location` Location
      - `country` string, required
      - `languages` string[], nullable
    - `proxy` 'none' | 'datacenter' | 'residential' | 'mobile' | 'auto'
    - `blockAssets` boolean: Skip images, fonts and media when the page is rendered in a browser. Faster.
    - `removeBase64Images` boolean: Strip inline base64 images out of the returned markdown and HTML.
    - `parsers` string[]: Which document parsers may run, for example `pdf`.
  - `webhook` WebhookConfig
    - `url` string, required
    - `events` string[]
    - `headers` object

## Response `200`

Successful Response

- object

## Other responses

- `422`: Validation Error

---

[API](https://skmtc.dev/snoopscan/apis/snoopscan-api.md) · [All operations](https://skmtc.dev/snoopscan/apis/snoopscan-api/llms.txt) · [OpenAPI document](https://skmtc.dev/snoopscan/apis/snoopscan-api/revisions/c278cdbcaf7b?raw)