---
title: "Starts a new crawl. You receive a `id` to track the progress. The operation may take 1-10 mins depending upon the site and depth and pages parameters."
method: POST
path: "/olostep/v1/crawls"
tags: ["Olostep"]
---

# Starts a new crawl. You receive a \`id\` to track the progress. The operation may take 1-10 mins depending upon the site and depth and pages parameters.

`POST /olostep/v1/crawls`

## Request

- Base URL: `https://api.orthogonal.com/mpp`
- URL: `https://api.orthogonal.com/mpp/olostep/v1/crawls`
- Auth: none declared
- TypeScript: <https://skmtc.dev/orthogonal/apis/orthogonal-api-marketplace/docs/olostep/v1/crawls/post.ts> (a function with typed inputs; its response is checked against the schema)

## Request body

- object
  - `start_url` string, required: The starting point of the crawl.
  - `max_pages` number, required: Maximum number of pages to crawl. Recommended for most use cases like crawling an entire website.
  - `include_urls` string[]: URL path patterns to include in the crawl using glob syntax. Defaults to `/**` which includes all URLs. Use patterns like `/blog/**` to crawl specific sections (e.g., only blog pages), `/products/*.html` for product pages, or multiple patterns for different sections. Supports standard glob features like * (any characters) and ** (recursive matching).
  - `exclude_urls` string[]: URL path names in glob pattern to exclude. For example: `/careers/**`. Excluded URLs will supersede included URLs.
  - `max_depth` number: Maximum depth of the crawl. Useful to extract only up to n-degree of links.
  - `include_external` boolean: Crawl first-degree external links.
  - `include_subdomain` boolean: Include subdomains of the website. `false` by default.
  - `search_query` string: An optional search query to find specific links and also sort the results by relevance.
  - `top_n` number: An optional number to only crawl the top N most relevant links on every page as per search query.
  - `webhook_url` string: An optional POST request endpoint called when this crawl is completed. The body of the request will be same as the response of this [`v1/crawls/{crawl_id}`](./info#response-created) endpoint.
  - `timeout` number: End the crawl after n seconds with the pages completed until then. May take ~10s extra from provided timeout.

## Response `200`

Successful response

## Other responses

- `402`: Payment Required (x402 / MPP / np)

## Changes

- **2026-11-08** `857e915717f4`: 1 breaking
  - added required request body
- **2026-08-25** `806e279c8d50`: 2 info
  - api tag `Olostep` added
  - api tag `Olostep API` removed

[Operation history](https://skmtc.dev/orthogonal/apis/orthogonal-api-marketplace/changes/olostep/v1/crawls/post.md)

---

[API](https://skmtc.dev/orthogonal/apis/orthogonal-api-marketplace.md) - [All operations](https://skmtc.dev/orthogonal/apis/orthogonal-api-marketplace/llms.txt) - [OpenAPI document](https://skmtc.dev/orthogonal/apis/orthogonal-api-marketplace/revisions/fc2e1d26c8eb?raw)