crawls

Create a crawl

Starts crawling a site from a starting URL.

post/v1/crawls

Request

  • Base URL: https://api.olostep.com
  • URL: https://api.olostep.com/v1/crawls
  • Auth: HTTP bearer

Request body

start_urlstring uri required

The URL to start crawling from.

include_urlsstring[]
exclude_urlsstring[]
max_pagesinteger
max_depthinteger
include_externalboolean
include_subdomainboolean
search_querystring
top_ninteger
webhookstring

Webhook URL, or {endpoint} object.

timeoutinteger
follow_robots_txtboolean

Response

Default Response

idstring required
object'crawl' required
statusstring required
creatednumber required
start_datestring
start_urlstring required
max_pagesnumber nullable
max_depthnumber nullable
exclude_urlsstring[]
include_urlsstring[]
include_externalboolean
search_querystring
top_nnumber
current_depthnumber
pages_countnumber
webhookstring nullable
webhook_urlstring nullable
follow_robots_txtboolean
credits_consumednumber nullable
cost_usdnumber nullable

Changes

No recorded changes to this operation across all 3 revisions of this API.