Check robots.txt policy for eight named AI crawler tokens

post/api/ai-crawler-check

Request

  • Base URL: https://actablesite.com
  • URL: https://actablesite.com/api/ai-crawler-check
  • Auth: none declared

Request body

urlstring uri required

Public HTTP or HTTPS website URL. Private and local-network targets are rejected.

Response

Crawler-policy check completed.

urlstring uri required
robotsUrlstring uri required
checkedAtstring date-time required
responseStatusinteger required
state'found' | 'missing' | 'unreadable' required

Changes

No recorded changes to this operation across all 1 revision of this API.