---
title: "Robots Txt"
method: GET
path: "/robots.txt"
---

# Robots Txt

`GET /robots.txt`

## Request

- Base URL: `https://www.oxfordledge.com` (the document declares no server; this is the origin it was published from)
- URL: `https://www.oxfordledge.com/robots.txt`
- Auth: none declared

Serve robots.txt for search engines + AI crawlers.

Posture (OWNER-ratified 2026-04-26 permissive-AI-crawl; reconciled #57
SEO-C3, 2026-06-28):
  - `*`: allow the public site, disallow API/admin/docs; Crawl-delay 10
    is the default brake for unknown/aggressive crawlers.
  - bingbot: same allow/disallow but NO Crawl-delay. Bing HONORS
    Crawl-delay (Google ignores it), so the global 10s capped Bing at
    ~1 page/10s — starving Bing's index AND ChatGPT Search, which rides
    Bing's index. Removing it lets Bing crawl the full corpus.
  - AI crawlers (GPTBot/OAI-SearchBot/ChatGPT-User/ClaudeBot/anthropic-ai/
    PerplexityBot/Google-Extended/Applebot-Extended/CCBot): explicitly
    WELCOMED per the permissive strategy, with the same content disallows
    and no Crawl-delay, so answer engines can read + cite the public
    corpus freely. (Each group repeats the disallows because robots.txt
    groups are NOT additive — a user-agent block inherits nothing from
    `*`, so omitting them would expose /api to that agent.)
  - SEO link-scrapers (Ahrefs/Semrush/MJ12/DotBot) + Baidu/Sogou: blocked.

## Response `200`

Successful Response

---

[API](https://skmtc.dev/oxfordledge/apis/oxford-ledge-api.md) · [All operations](https://skmtc.dev/oxfordledge/apis/oxford-ledge-api/llms.txt) · [OpenAPI document](https://skmtc.dev/oxfordledge/apis/oxford-ledge-api/revisions/e832a1260f0b?raw)