SCscrunch
crawl-readiness

Get AI Crawl Readiness

Changed on

Whether AI crawlers can reach one of the brand's domains, based on the brand's most recent finished crawl of it. These are the same signals the AI Access card shows in the Scrunch dashboard.

Pass domain to choose which of the brand's domains to describe; omit it for the primary website. The response is null when the brand owns the domain but no crawl of it has finished yet; requesting a domain the brand does not own returns 404.

Reading the fields:

  • robots_txt_present: whether a robots.txt was found. False means permissive, not blocked: with no robots.txt, crawlers are allowed by default.
  • is_blocking_bots: the site turns crawlers away, derived from the share of pages that answered HTTP 403 across the whole crawl. This is a server-level verdict and is unrelated to blocked_pages, which is about robots.txt.
  • is_spa: most pages render client-side, so a crawler that does not run JavaScript reads little or nothing.
  • blocked_pages: pages robots.txt blocks, with the AI assistant families blocked from each. This is a sample, not a complete list: only the 50 shallowest pages of the crawl are checked. pages_truncated is true when the crawl held more than that, so an empty or short list does not mean nothing else on the site is blocked.
get/{brand_id}/crawl-readiness

Request

Path parameters

brand_idinteger required

The unique identifier for the brand.

Query parameters

domainstring nullable

Which of the brand's domains to describe. Omit for the primary website.

Response

Successful Response. null when the brand owns the domain but no crawl of it has finished yet.

domainstring required

The domain the readiness signals describe.

robots_txt_presentboolean required

Whether a robots.txt was found for the domain. False means permissive, not blocked: with no robots.txt, crawlers are allowed by default.

is_spaboolean required

Most pages render client-side, so a crawler that does not run JavaScript reads little or nothing.

is_blocking_botsboolean required

The site turns crawlers away at the server level, derived from the share of pages that answered HTTP 403 across the whole crawl.

pages_truncatedboolean required

True when the crawl held more pages than the 50-page scan cap, so blocked_pages is a sample.

Changes