/robots.txt for a domain's hosted subdomain (see backend/agent_desk/robots.py for the rule sets and why they are what they are). The edge worker proxies here on an R2 miss for /robots.txt, which means a customer who publishes their own object at that key still wins.

A domain without an enabled desk gets the x-desk-robots: disabled sentinel and the edge keeps its pre-existing behavior for the path. That is the whole reason this is a control-plane call rather than a string the worker could assemble by itself: "is this host a desk" is control-plane state, and a desk-shaped robots.txt appearing on a hosted-content host that has no desk would be publishing crawl policy for a surface that doesn't exist.