SPspider

AI crawl

Changed on

Crawl a website with a natural-language prompt guiding extraction. Requires a valid API key for an account with an active AI subscription plan; AI usage is billed separately from credits.

post/ai/crawl

Request

  • Base URL
    https://api.spider.cloud
  • URL
    https://api.spider.cloud/ai/crawl
  • Auth
    none declared

Request body

urlstring
smart_modeboolean
authorizationstring
cookiesstring
refresh_tokenstring
hard_limitinteger
budgetobject
store_databoolean
use_chromeboolean
root_selectorstring
exclude_selectorstring
run_in_backgroundboolean
request'http' | 'chrome' | 'SmartMode'
depthinteger
encodingstring
localestring
request_timeoutinteger
delayinteger
stealthboolean
headersobject
viewportobject
device'mobile' | 'tablet' | 'desktop'
readabilityboolean
subdomainsboolean
tldboolean
user_agentstring
blackliststring[]
whiteliststring[]
full_resourcesboolean
metadataboolean
fingerprintboolean
anti_botboolean
respect_robotsboolean
chunking_alg'No' | 'ByWords' | 'ByLines' | 'ByCharacterLength' | 'BySentence'
skip_config_checksboolean
textstring
website_limitinteger
sitemapboolean
sitemap_pathstring
country_codestring
wait_forobject
execution_scriptsobject
disable_interceptboolean
disable_first_party_stylesheetsboolean

Disable the first-party stylesheet allow when block_stylesheets is on. Default false — first-party CSS still passes so SPAs hydrate. Set true for strict block-all-CSS bandwidth-minimal crawls.

disable_first_party_javascriptboolean

Disable the first-party JavaScript allow through downstream blockers (intercept manager / adblock / blocklists). Default false. Set true to enforce blocklists strictly even on first-party scripts.

disable_first_party_visualsboolean

Disable the first-party visual allow when block_visuals is on. Default false. Set true for strictly bandwidth-minimal crawls that drop ALL visuals regardless of origin.

external_domainsstring[]
css_extraction_mapobject
scrollinteger
automation_scriptsobject
return_headersboolean
return_cookiesboolean
return_page_linksboolean
return_json_databoolean
return_embeddingsboolean
virtual_displayboolean
redirect_policy'Loose' | 'Strict'
preserve_hostboolean
filter_output_imagesboolean
filter_output_svgboolean
filter_output_main_onlyboolean
remote_proxystring
concurrency_limitinteger
promptstring required
extraction_schemaobject
cleaning_intent'extraction' | 'action' | 'general'

Response

Successful AI crawl

Changes