---
title: "Facet the AI-readiness (llms.txt) dataset"
method: GET
path: "/datasets/llms/facets"
tags: ["Datasets"]
---

# Facet the AI-readiness (llms.txt) dataset

`GET /datasets/llms/facets`

## Request

- Base URL: `https://api.crawlora.net/api/v1`
- URL: `https://api.crawlora.net/api/v1/datasets/llms/facets`
- Auth: API key in header `x-api-key`

Returns distribution counts over the AI-readiness census (dataset id enum value `llms`), honoring the same filters as search — adoption by producer, generator, TLD or category, and which AI crawlers robots.txt blocks. Facet enum: `state`, `source_kind`, `generator`, `bot`, `policy`, `tld`, `category`, `robots_state`, `robots_default`, `blocked_crawler`, `partial_crawler`, `content_signal_train`, `full_state`, `home_status`. The filter enums are the same as search. Counts include `blocked` and `error` probes, which are unknown rather than absent; filter `state=present` or `state=absent` to count only sites that were established either way.

## Query parameters

- `facet` 'state' | 'source_kind' | 'generator' | 'bot' | 'policy' | 'tld' | 'category' | 'robots_state' | 'robots_default' | 'blocked_crawler' | 'partial_crawler' | 'content_signal_train' | 'full_state' | 'home_status', required
- `q` string
- `text` string
- `state` 'present' | 'absent' | 'blocked' | 'error' | 'skipped'
- `valid` boolean
- `source_kind` 'shopify_agent_template' | 'seo_plugin' | 'platform_default' | 'parked' | 'custom'
- `generator` 'yoast' | 'aioseo' | 'rankmath' | 'wix' | 'shopify' | 'parked'
- `bot` string
- `min_links` integer
- `mentions_mcp` boolean
- `mentions_openapi` boolean
- `placeholder` boolean
- `injection` boolean
- `full_valid` boolean
- `robots_state` 'present' | 'absent' | 'blocked' | 'error' | 'skipped'
- `robots_default` 'allowed' | 'partial' | 'blocked'
- `blocks_all` string[]
- `blocks_any` string[]
- `not_blocked` string[]
- `all_ai_blocked` boolean
- `min_train_blocked` integer
- `content_signal_train` 'yes' | 'no'
- `tld` string
- `category` string
- `reachable` boolean
- `home_blocked` boolean
- `is_infrastructure` boolean
- `run_id` string

## Headers

- `x-api-key` string, required

## Response `200`

OK

- DatasetsLlmsFacetResponseDoc
  - `code` integer
  - `data` DatasetsLlmsFacetResponse
    - `dataset` string
    - `facet` string
    - `items` EsLlmsDatasetFacetItem[]
      - `count` integer
      - `value` string
  - `msg` string

## Other responses

- `400`: Bad Request
- `429`: Too Many Requests
- `500`: Internal Server Error

---

[API](https://skmtc.dev/crawlora/apis/crawlora-api.md) · [All operations](https://skmtc.dev/crawlora/apis/crawlora-api/llms.txt) · [OpenAPI document](https://skmtc.dev/crawlora/apis/crawlora-api/revisions/028417aa35bb?raw)