---
title: "Get AI Crawl Readiness"
method: GET
path: "/{brand_id}/crawl-readiness"
tags: ["crawl-readiness"]
---

# Get AI Crawl Readiness

`GET /{brand_id}/crawl-readiness`

## Request

- Base URL: `https://api.scrunchai.com/v1`
- URL: `https://api.scrunchai.com/v1/{brand_id}/crawl-readiness`
- Auth: HTTP bearer
- TypeScript: <https://skmtc.dev/scrunch/apis/scrunch-data-api/docs/:brand_id/crawl-readiness/get.ts> (a function with typed inputs; its response is checked against the schema)

Whether AI crawlers can reach one of the brand's domains, based on the brand's most recent finished crawl of it. These are the same signals the AI Access card shows in the Scrunch dashboard.

Pass `domain` to choose which of the brand's domains to describe; omit it for the primary website. The response is `null` when the brand owns the domain but no crawl of it has finished yet; requesting a domain the brand does not own returns `404`.

Reading the fields:

- `robots_txt_present`: whether a robots.txt was found. **False means permissive**, not blocked: with no robots.txt, crawlers are allowed by default.
- `is_blocking_bots`: the site turns crawlers away, derived from the share of pages that answered HTTP 403 across the whole crawl. This is a server-level verdict and is unrelated to `blocked_pages`, which is about robots.txt.
- `is_spa`: most pages render client-side, so a crawler that does not run JavaScript reads little or nothing.
- `blocked_pages`: pages robots.txt blocks, with the AI assistant families blocked from each. **This is a sample, not a complete list**: only the 50 shallowest pages of the crawl are checked. `pages_truncated` is true when the crawl held more than that, so an empty or short list does not mean nothing else on the site is blocked.

## Path parameters

- `brand_id` integer, required: The unique identifier for the brand.

## Query parameters

- `domain` string, nullable: Which of the brand's domains to describe. Omit for the primary website.

## Response `200`

Successful Response. `null` when the brand owns the domain but no crawl of it has finished yet.

- CrawlReadinessModel: AI crawl readiness signals for one of the brand's domains, from its most recent finished crawl.
  - `domain` string, required: The domain the readiness signals describe.
  - `robots_txt_present` boolean, required: Whether a robots.txt was found for the domain. False means permissive, not blocked: with no robots.txt, crawlers are allowed by default.
  - `is_spa` boolean, required: Most pages render client-side, so a crawler that does not run JavaScript reads little or nothing.
  - `is_blocking_bots` boolean, required: The site turns crawlers away at the server level, derived from the share of pages that answered HTTP 403 across the whole crawl.
  - `blocked_pages` BlockedPageModel[], required: Pages robots.txt blocks from at least one AI assistant family. A sample from the 50 shallowest pages of the crawl, not a complete list.
    - `path` string, required: The page's path on the site, for example `/pricing`.
    - `url` string, required: The page's full URL.
    - `blocked_by` string[], required: The AI assistant families (OpenAI, Meta, Perplexity, Gemini, Claude) that robots.txt blocks from this page. Always non-empty for a page in this list.
  - `pages_truncated` boolean, required: True when the crawl held more pages than the 50-page scan cap, so blocked_pages is a sample.

## Other responses

- `404`: Brand not found, or the brand does not own the requested domain.
- `422`: Validation Error

## Changes

- **2026-11-03** `e37811b6bc07`: 1 info
  - endpoint added

[Operation history](https://skmtc.dev/scrunch/apis/scrunch-data-api/changes/:brand_id/crawl-readiness/get.md)

---

[API](https://skmtc.dev/scrunch/apis/scrunch-data-api.md) - [All operations](https://skmtc.dev/scrunch/apis/scrunch-data-api/llms.txt) - [OpenAPI document](https://skmtc.dev/scrunch/apis/scrunch-data-api/revisions/3c6aa69a6595?raw)