---
title: "Robots Txt"
method: GET
path: "/robots.txt"
tags: ["meta"]
---

# Robots Txt

`GET /robots.txt`

## Request

- Base URL: `https://api.getcheapgames.com` (the document declares no server; this is the origin it was published from)
- URL: `https://api.getcheapgames.com/robots.txt`
- Auth: none declared

The `/*/search` wildcard is load-bearing.

Every real path is country-prefixed (`/gb/search?q=...`), and a `Disallow`
is a PREFIX match from the start of the path — so the plain `/search` rule
this shipped with matched nothing at all, and internal search was fully
crawlable in all 39 markets. The page's own `noindex` only exists once a
crawler has executed the JavaScript, by which point the crawl has been paid
for. The unprefixed rule is kept below for the bare form.

The catalogue API is NOT disallowed, and must never be again. Google and
Bing render the app, and a renderer honours robots.txt for every fetch the
page makes: with `Disallow: /api/` every call the app renders from was
refused, so the page they indexed had no title, no description, no
canonical, no hreflang, no JSON-LD and no content — the shell's tags are
removed on boot (main.tsx) and nothing replaced them. Measured by loading
the shell with /api/ refused: an empty page, against 1,073 words and 47
links with it allowed. The JSON is kept out of the index by
`X-Robots-Tag: noindex` instead (main.py), which a renderer still fetches
past. Only the endpoints that are not reads stay disallowed: the affiliate
redirect, the social callbacks, and search.

No `Allow: /` line. Everything is allowed by default, and a parser that
takes the FIRST matching rule rather than the longest — Python's own
`robotparser`, and crawlers built on it — read the `Allow: /` that used to
head this group as permission for every path, disallowed ones included.
And no blank line inside the group: the original robots.txt convention
ends a record at one, so to those parsers every rule after it belonged to
no user-agent at all.

## Response `200`

Successful Response

---

[API](https://skmtc.dev/getcheapgames/apis/getcheapgames-api.md) · [All operations](https://skmtc.dev/getcheapgames/apis/getcheapgames-api/llms.txt) · [OpenAPI document](https://skmtc.dev/getcheapgames/apis/getcheapgames-api/revisions/57333a690301?raw)