---
title: "Robots Txt By Domain"
method: GET
path: "/internal/agent-desk/robots-txt"
tags: ["internal"]
---

# Robots Txt By Domain

`GET /internal/agent-desk/robots-txt`

## Request

- Base URL: `https://api-v1.unusual.ai` (the document declares no server; this is the origin it was published from)
- URL: `https://api-v1.unusual.ai/internal/agent-desk/robots-txt`
- Auth: none declared

``/robots.txt`` for a domain's hosted subdomain (see
backend/agent_desk/robots.py for the rule sets and why they are what they
are). The edge worker proxies here on an R2 miss for ``/robots.txt``, which
means a customer who publishes their own object at that key still wins.

A domain without an enabled desk gets the ``x-desk-robots: disabled``
sentinel and the edge keeps its pre-existing behavior for the path. That is
the whole reason this is a control-plane call rather than a string the
worker could assemble by itself: "is this host a desk" is control-plane
state, and a desk-shaped robots.txt appearing on a hosted-content host that
has no desk would be publishing crawl policy for a surface that doesn't
exist.

## Query parameters

- `domain_id` string, required: The domain ID resolved by the edge worker

## Headers

- `x-internal-key` string, nullable
- `x-internal-token` string, nullable

## Response `200`

Successful Response

- unknown

## Other responses

- `422`: Validation Error

---

[API](https://skmtc.dev/unusual/apis/dashboard-server.md) - [All operations](https://skmtc.dev/unusual/apis/dashboard-server/llms.txt) - [OpenAPI document](https://skmtc.dev/unusual/apis/dashboard-server/revisions/89f716930ead?raw)