---
title: "Robots Txt"
method: HEAD
path: "/robots.txt"
---

# Robots Txt

`HEAD /robots.txt`

## Request

- Base URL: `https://agentforge.sciscale.org` (the document declares no server; this is the origin it was published from)
- URL: `https://agentforge.sciscale.org/robots.txt`
- Auth: none declared

Permissive robots.txt — we WANT to be indexed (including by GPTBot
/ ChatGPT-User / Claude-Web for LLM-SEO purposes; we do NOT block them).
The Disallow list covers:
  - /api/*           — JSON endpoints, useless to index
  - /generate        — htmx fragment endpoint, useless to index
  - /login, /logout  — auth flow, never useful in search results
  - /account/        — per-user pages, never crawlable anyway
  - /history/        — per-user history, never crawlable anyway

Points crawlers at the sitemap so they can discover blog posts
without depending on internal-link crawl depth.

## Response `200`

Successful Response

---

[API](https://skmtc.dev/sciscale/apis/agentforge.md) · [All operations](https://skmtc.dev/sciscale/apis/agentforge/llms.txt) · [OpenAPI document](https://skmtc.dev/sciscale/apis/agentforge/revisions/402c3d267f68?raw)