Robots Txt

Permissive robots.txt — we WANT to be indexed (including by GPTBot / ChatGPT-User / Claude-Web for LLM-SEO purposes; we do NOT block them). The Disallow list covers:

  • /api/* — JSON endpoints, useless to index
  • /generate — htmx fragment endpoint, useless to index
  • /login, /logout — auth flow, never useful in search results
  • /account/ — per-user pages, never crawlable anyway
  • /history/ — per-user history, never crawlable anyway

Points crawlers at the sitemap so they can discover blog posts without depending on internal-link crawl depth.

get/robots.txt

Request

  • Base URL: https://agentforge.sciscale.org (the document declares no server; this is the origin it was published from)
  • URL: https://agentforge.sciscale.org/robots.txt
  • Auth: none declared

Response

Successful Response

Changes

No recorded changes to this operation across all 1 revision of this API.