---
title: "AI Crawler Permission Audit — signed robots.txt / llms.txt / ai.txt verdict (keyless x402)"
method: GET
path: "/buy/robots"
tags: ["robots-txt", "ai-crawler-policy", "crawler-permission-audit", "llms-txt", "ai-txt", "tdm-opt-out", "eu-ai-act-compliance", "ai-bot-blocking", "training-data-consent", "verifiable-policy-snapshot", "provenance", "attestation", "keyless", "agent-native", "data-feed"]
---

# AI Crawler Permission Audit — signed robots.txt / llms.txt / ai.txt verdict (keyless x402)

`GET /buy/robots`

## Request

- Base URL: `https://network.mercury-hq.com`
- URL: `https://network.mercury-hq.com/buy/robots`
- Auth: none declared
- TypeScript: <https://skmtc.dev/mercury-hq/apis/mercury-x402-storefront/docs/buy/robots/get.ts> (a function with typed inputs; its response is checked against the schema)

Domain → signed, timestamped per-AI-crawler allow/block audit from robots.txt + llms.txt + ai.txt (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider, …). Deterministic, no LLM. EU-AI-Act / TDM opt-out evidence: the signed verdict proves the crawler policy as it stood at fetch time.

## Query parameters

- `domain` string, required: domain or URL to audit (e.g. example.com or https://example.com/page); only the origin is used
- `path` string: optional path to evaluate the verdict for (default '/', the whole-site question)

## Response `200`

Delivered after the x402 payment settles on Base mainnet.

- object
  - `ok` boolean, required: true on success; false on an honest failure (still delivered, never charged for a stub)
  - `url` string, required: origin audited (https://host)
  - `status` integer: HTTP status of the robots.txt fetch
  - `text` string: canonical JSON the signed attestation is computed over (data, keys sorted)
  - `fetchedAt` string: ISO-8601 fetch time — the 'as-of' moment the signed policy snapshot proves
  - `data` object: the structured audit
    - `version` string
    - `host` string
    - `origin` string
    - `evaluatedPath` string
    - `policies` object: raw state of each crawler-policy file at fetch time
      - `robots.txt` object
      - `llms.txt` object
      - `ai.txt` object
    - `verdicts` object[]: per-AI-agent allow/block verdict for evaluatedPath, with the decisive rule cited
      - `agent` string
      - `vendor` string
      - `purpose` string
      - `verdict` 'allowed' | 'blocked'
      - `reason` string
      - `decidedBy` string
      - `rule` string, nullable
    - `summary` object
      - `allowed` integer
      - `blocked` integer
      - `blockedAgents` string[]
      - `sitemaps` string[]
      - `hasLlmsTxt` boolean
      - `hasAiTxt` boolean
      - `hasRobotsTxt` boolean
  - `error` string: present only when ok:false

## Other responses

- `402`: Payment Required — retry with an x402-signed payment (e.g. x402-fetch). Terms are in the challenge body (x402 v1).

---

[API](https://skmtc.dev/mercury-hq/apis/mercury-x402-storefront.md) - [All operations](https://skmtc.dev/mercury-hq/apis/mercury-x402-storefront/llms.txt) - [OpenAPI document](https://skmtc.dev/mercury-hq/apis/mercury-x402-storefront/revisions/211aa7db24ae?raw)