---
title: "Public crawler policy for search and answer engines, including preferred crawl entrypoints and allowed user agents."
method: GET
path: "/.well-known/ai-crawler-policy.json"
tags: ["Agent Docs"]
---

# Public crawler policy for search and answer engines, including preferred crawl entrypoints and allowed user agents.

`GET /.well-known/ai-crawler-policy.json`

## Request

- Base URL: `https://huggingbay.xyz`
- URL: `https://huggingbay.xyz/.well-known/ai-crawler-policy.json`
- Auth: none declared

Public crawler policy for search and answer engines, including preferred crawl entrypoints and allowed user agents.

**Inputs.** No path or query fields are required. No request body is required.

**Outputs.** See each status below for its media type, schema and headers. A missing record or unavailable capability is not a successful empty result.

**Errors.** Read the status and Content-Type before decoding. Document, image, redirect and file routes need not return JSON errors.

**Rate limits.** Public read budgets apply by client IP and route group. This operation can have a smaller route-specific budget. Read Retry-After on 429 or retryable 503 responses.

## Response `200`

AI crawler policy

- AiCrawlerPolicy
  - `schemaVersion` 'hugging-bay.ai-crawler-policy.v1', required
  - `generatedAt` string, required
  - `baseUrl` string, required
  - `robotsUrl` string, required
  - `answerEngineManifestUrl` string, required
  - `botAllowlistUrl` string, required
  - `llmsTxtUrl` string, required
  - `llmsFullTxtUrl` string, required
  - `policy` object, required
  - `preferredCrawlEntrypoints` string[], required
  - `preferredCitationTargets` string[], required
  - `userAgents` object[], required
  - `rateGuidance` object, required

---

[API](https://skmtc.dev/huggingbay/apis/hugging-bay-public-api.md) · [All operations](https://skmtc.dev/huggingbay/apis/hugging-bay-public-api/llms.txt) · [OpenAPI document](https://skmtc.dev/huggingbay/apis/hugging-bay-public-api/revisions/ea7ba26b1e27?raw)