Robots Txt

The /*/search wildcard is load-bearing.

Every real path is country-prefixed (/gb/search?q=...), and a Disallow is a PREFIX match from the start of the path — so the plain /search rule this shipped with matched nothing at all, and internal search was fully crawlable in all 39 markets. The page's own noindex only exists once a crawler has executed the JavaScript, by which point the crawl has been paid for. The unprefixed rule is kept below for the bare form.

The catalogue API is NOT disallowed, and must never be again. Google and Bing render the app, and a renderer honours robots.txt for every fetch the page makes: with Disallow: /api/ every call the app renders from was refused, so the page they indexed had no title, no description, no canonical, no hreflang, no JSON-LD and no content — the shell's tags are removed on boot (main.tsx) and nothing replaced them. Measured by loading the shell with /api/ refused: an empty page, against 1,073 words and 47 links with it allowed. The JSON is kept out of the index by X-Robots-Tag: noindex instead (main.py), which a renderer still fetches past. Only the endpoints that are not reads stay disallowed: the affiliate redirect, the social callbacks, and search.

No Allow: / line. Everything is allowed by default, and a parser that takes the FIRST matching rule rather than the longest — Python's own robotparser, and crawlers built on it — read the Allow: / that used to head this group as permission for every path, disallowed ones included. And no blank line inside the group: the original robots.txt convention ends a record at one, so to those parsers every rule after it belonged to no user-agent at all.