Robots Txt

Serve robots.txt for search engines + AI crawlers.

Posture (OWNER-ratified 2026-04-26 permissive-AI-crawl; reconciled #57 SEO-C3, 2026-06-28):

  • *: allow the public site, disallow API/admin/docs; Crawl-delay 10 is the default brake for unknown/aggressive crawlers.
  • bingbot: same allow/disallow but NO Crawl-delay. Bing HONORS Crawl-delay (Google ignores it), so the global 10s capped Bing at ~1 page/10s — starving Bing's index AND ChatGPT Search, which rides Bing's index. Removing it lets Bing crawl the full corpus.
  • AI crawlers (GPTBot/OAI-SearchBot/ChatGPT-User/ClaudeBot/anthropic-ai/ PerplexityBot/Google-Extended/Applebot-Extended/CCBot): explicitly WELCOMED per the permissive strategy, with the same content disallows and no Crawl-delay, so answer engines can read + cite the public corpus freely. (Each group repeats the disallows because robots.txt groups are NOT additive — a user-agent block inherits nothing from *, so omitting them would expose /api to that agent.)
  • SEO link-scrapers (Ahrefs/Semrush/MJ12/DotBot) + Baidu/Sogou: blocked.
get/robots.txt

Request

  • Base URL: https://www.oxfordledge.com (the document declares no server; this is the origin it was published from)
  • URL: https://www.oxfordledge.com/robots.txt
  • Auth: none declared

Response

Successful Response

Changes

No recorded changes to this operation across all 1 revision of this API.