---
title: "Machine-readable robots and AI crawler policy with public/private boundaries and preferred discovery URLs."
method: GET
path: "/robots.json"
tags: ["Agent Docs"]
---

# Machine-readable robots and AI crawler policy with public/private boundaries and preferred discovery URLs.

`GET /robots.json`

## Request

- Base URL: `https://huggingbay.xyz`
- URL: `https://huggingbay.xyz/robots.json`
- Auth: none declared

Machine-readable robots and AI crawler policy with public/private boundaries and preferred discovery URLs.

**Inputs.** No path or query fields are required. No request body is required.

**Outputs.** See each status below for its media type, schema and headers. A missing record or unavailable capability is not a successful empty result.

**Errors.** Read the status and Content-Type before decoding. Document, image, redirect and file routes need not return JSON errors.

**Rate limits.** Public read budgets apply by client IP and route group. This operation can have a smaller route-specific budget. Read Retry-After on 429 or retryable 503 responses.

## Response `200`

Machine-readable robots policy

- RobotsJson
  - `schemaVersion` 'hugging-bay.robots-json.v1', required
  - `generatedAt` string, required
  - `baseUrl` string, required
  - `textRobotsUrl` string, required
  - `status` string, required
  - `publicPolicy` object, required
  - `discovery` object, required
  - `userAgents` object[], required
  - `providerGroups` object[], required
  - `allowPublicPaths` string[]
  - `protectPrivatePaths` string[]
  - `preferredCrawlEntrypoints` string[], required
  - `preferredCitationTargets` string[], required
  - `expectedHeaders` object, required
  - `rateGuidance` object, required

---

[API](https://skmtc.dev/huggingbay/apis/hugging-bay-public-api.md) · [All operations](https://skmtc.dev/huggingbay/apis/hugging-bay-public-api/llms.txt) · [OpenAPI document](https://skmtc.dev/huggingbay/apis/hugging-bay-public-api/revisions/63c474bf0fe4?raw)