The served robots.txt, corrected: * group rules copied into every named group that lacked them, a Sitemap line added when the file never named the sitemap it serves, a minimal replacement when the served file was HTML. Every change is listed at the top of the file

get/api/fix/robots

Request

  • Base URL: https://crawlcheck.io
  • URL: https://crawlcheck.io/api/fix/robots
  • Auth: one of:
    • API key in header x-crawlcheck-key
    • none

Query parameters

domainstring required
format'json'

The notes as data instead of the file

Response

text/plain robots.txt, or JSON when format=json

Changes

No recorded changes to this operation across all 1 revision of this API.