---
title: "Compare robots.txt policy with synthetic AI search-crawler responses"
method: POST
path: "/api/cloudflare-ai-crawler-check"
---

# Compare robots.txt policy with synthetic AI search-crawler responses

`POST /api/cloudflare-ai-crawler-check`

## Request

- Base URL: `https://actablesite.com`
- URL: `https://actablesite.com/api/cloudflare-ai-crawler-check`
- Auth: none declared

Sends bounded public requests using three documented search-crawler user-agent tokens. The requests do not originate from provider IP ranges, authenticate a provider bot, inspect a Cloudflare account, identify the cause of a response, or prove crawling, indexing, citation, ranking, referral traffic, or revenue.

## Request body

- WebsiteRequest
  - `url` string, uri, required: Public HTTP or HTTPS website URL. Private and local-network targets are rejected.

## Response `200`

Robots policy and synthetic-response comparison completed.

- CloudflareCrawlerCheckResponse
  - `url` string, uri, required
  - `checkedAt` string, date-time, required
  - `robotsUrl` string, uri, required
  - `robotsStatus` integer, required
  - `robotsState` 'found' | 'missing' | 'unreadable', required
  - `cloudflareObserved` boolean, required
  - `comparisons` CloudflareCrawlerComparison[], required
    - `agent` string, required
    - `provider` string, required
    - `purpose` string, required
    - `category` 'search', required
    - `policy` CrawlerResult, required
      - `agent` string, required
      - `provider` string, required
      - `purpose` string, required
      - `category` 'search' | 'training' | 'ads' | 'user', required
      - `allowed` boolean, nullable, required
      - `matchedDirective` string, nullable, required
      - `matchedGroup` string, nullable
      - `evidence` string, required
    - `synthetic` SyntheticCrawlerResult, required
      - `agent` string, required
      - `provider` string, required
      - `purpose` string, required
      - `status` integer, nullable, required
      - `state` 'reachable' | 'restricted' | 'unavailable', required
      - `cloudflareObserved` boolean, required
      - `evidence` string, required
  - `limits` string, required

## Other responses

- `400`: The request did not contain a valid public HTTP or HTTPS website URL.

---

[API](https://skmtc.dev/actablesite/apis/actablesite-public-api.md) · [All operations](https://skmtc.dev/actablesite/apis/actablesite-public-api/llms.txt) · [OpenAPI document](https://skmtc.dev/actablesite/apis/actablesite-public-api/revisions/c36ab479a920?raw)