DAdatabuddy
Websites

Check robots.txt rules for AI crawlers

Reads the website's robots.txt and reports, for each AI crawler user agent, whether it is allowed, partly blocked, or blocked, and since when a blocked crawler has been blocked. Requires website read permission.

post/websites/checkAiRobots

Request

  • Base URL
    https://api.databuddy.cc
  • URL
    https://api.databuddy.cc/websites/checkAiRobots
  • Auth
    API key in header x-api-key

Request body

websiteIdstring required
userAgentsstring[] required

Response

OK

hasRobotsTxtboolean required
accessstring[] required
blockedSincestring[] required

Changes

No recorded changes to this operation across all 1 revision of this API.