---
title: "Get Ai Crawl Readiness"
method: GET
path: "/{brand_id}/crawl-readiness"
tags: ["crawl-readiness"]
---

# Get Ai Crawl Readiness

`GET /{brand_id}/crawl-readiness`

## Request

- Base URL: `https://api.scrunchai.com/v1`
- URL: `https://api.scrunchai.com/v1/{brand_id}/crawl-readiness`
- Auth: HTTP bearer

Whether AI crawlers can reach one of the brand's domains, from the brand's most recent finished crawl of it — the same signals the AI Access card shows.

Pass `domain` to choose which of the brand's domains to describe; omit it for the primary website. `null` when the brand owns that domain but no crawl of it has finished yet; 404 for a domain the brand does not own.

Reading the fields:

- `robots_txt_present` — whether a robots.txt was found. **False means permissive**, not blocked: with no robots.txt, crawlers are allowed by default.
- `is_blocking_bots` — the site turns crawlers away, derived from the share of pages that answered HTTP 403 across the whole crawl. This is a server-level verdict and is unrelated to `blocked_pages`, which is about robots.txt.
- `is_spa` — most pages render client-side, so a crawler that does not run JavaScript reads little or nothing.
- `blocked_pages` — pages robots.txt blocks, with the AI assistant families blocked from each. **This is a sample, not a complete list**: only the 50 shallowest pages of the crawl are checked. `pages_truncated` is true when the crawl held more than that, so an empty or short list does not mean nothing else on the site is blocked.

## Path parameters

- `brand_id` integer, required

## Query parameters

- `domain` string, nullable

## Response `200`

Successful Response

- CrawlReadinessModel: Everything the Crawl Readiness card renders, resolved over the brand's current mapped site.
  - `domain` string, required
  - `robots_txt_present` boolean, required
  - `is_spa` boolean, required
  - `is_blocking_bots` boolean, required
  - `blocked_pages` BlockedPageModel[], required
    - `path` string, required
    - `url` string, required
    - `blocked_by` string[], required
  - `pages_truncated` boolean, required

## Other responses

- `422`: Validation Error

## Changes

- **2026-09-08** `86b9d11e042c`: 1 info
  - endpoint added

[Operation history](https://skmtc.dev/scrunchai/apis/scrunch-data-api/changes/:brand_id/crawl-readiness/get.md)

---

[API](https://skmtc.dev/scrunchai/apis/scrunch-data-api.md) - [All operations](https://skmtc.dev/scrunchai/apis/scrunch-data-api/llms.txt) - [OpenAPI document](https://skmtc.dev/scrunchai/apis/scrunch-data-api/revisions/de4fc679600a?raw)