crawl-readiness

Get Ai Crawl Readiness

Changed on

Whether AI crawlers can reach one of the brand's domains, from the brand's most recent finished crawl of it — the same signals the AI Access card shows.

Pass domain to choose which of the brand's domains to describe; omit it for the primary website. null when the brand owns that domain but no crawl of it has finished yet; 404 for a domain the brand does not own.

Reading the fields:

  • robots_txt_present — whether a robots.txt was found. False means permissive, not blocked: with no robots.txt, crawlers are allowed by default.
  • is_blocking_bots — the site turns crawlers away, derived from the share of pages that answered HTTP 403 across the whole crawl. This is a server-level verdict and is unrelated to blocked_pages, which is about robots.txt.
  • is_spa — most pages render client-side, so a crawler that does not run JavaScript reads little or nothing.
  • blocked_pages — pages robots.txt blocks, with the AI assistant families blocked from each. This is a sample, not a complete list: only the 50 shallowest pages of the crawl are checked. pages_truncated is true when the crawl held more than that, so an empty or short list does not mean nothing else on the site is blocked.
get/{brand_id}/crawl-readiness

Request

  • Base URL
    https://api.scrunchai.com/v1
  • Auth
    HTTP bearer

Path parameters

brand_idinteger required

Query parameters

domainstring nullable

Response

Successful Response

domainstring required
robots_txt_presentboolean required
is_spaboolean required
is_blocking_botsboolean required
pages_truncatedboolean required

Changes