---
title: "Check robots.txt policy for three named OpenAI crawler tokens"
method: POST
path: "/api/openai-robots-check"
---

# Check robots.txt policy for three named OpenAI crawler tokens

`POST /api/openai-robots-check`

## Request

- Base URL: `https://actablesite.com`
- URL: `https://actablesite.com/api/openai-robots-check`
- Auth: none declared

## Request body

- WebsiteRequest
  - `url` string, uri, required: Public HTTP or HTTPS website URL. Private and local-network targets are rejected.

## Response `200`

OpenAI crawler-policy check completed.

- CrawlerCheckResponse
  - `url` string, uri, required
  - `robotsUrl` string, uri, required
  - `checkedAt` string, date-time, required
  - `responseStatus` integer, required
  - `state` 'found' | 'missing' | 'unreadable', required
  - `crawlers` CrawlerResult[], required
    - `agent` string, required
    - `provider` string, required
    - `purpose` string, required
    - `category` 'search' | 'training' | 'ads' | 'user', required
    - `allowed` boolean, nullable, required
    - `matchedDirective` string, nullable, required
    - `matchedGroup` string, nullable
    - `evidence` string, required

## Other responses

- `400`: The request did not contain a valid public HTTP or HTTPS website URL.

---

[API](https://skmtc.dev/actablesite/apis/actablesite-public-api.md) · [All operations](https://skmtc.dev/actablesite/apis/actablesite-public-api/llms.txt) · [OpenAPI document](https://skmtc.dev/actablesite/apis/actablesite-public-api/revisions/c36ab479a920?raw)