Facet the AI-readiness (llms.txt) dataset
Returns distribution counts over the AI-readiness census (dataset id enum value llms), honoring the same filters as search — adoption by producer, generator, TLD or category, and which AI crawlers robots.txt blocks. Facet enum: state, source_kind, generator, bot, policy, tld, category, robots_state, robots_default, blocked_crawler, partial_crawler, content_signal_train, full_state, home_status. The filter enums are the same as search. Counts include blocked and error probes, which are unknown rather than absent; filter state=present or state=absent to count only sites that were established either way.
Request
- Base URL: https://api.crawlora.net/api/v1
- URL: https://api.crawlora.net/api/v1/datasets/llms/facets
- Auth: API key in header x-api-key
Query parameters
Facet enum: state, source_kind, generator, bot, policy, tld, category, robots_state, robots_default, blocked_crawler, partial_crawler, content_signal_train, full_state, home_status
Substring match on the site domain, max 256 characters
Full-text match on the /llms.txt title and summary, max 256 characters
/llms.txt state. Enum: present, absent, blocked, error, skipped. blocked and error mean unknown, not absent
true keeps only sites serving a valid /llms.txt, false keeps the rest
Producer of a valid /llms.txt. Enum: shopify_agent_template, seo_plugin, platform_default, parked, custom
Generator detected in a valid /llms.txt. Enum: yoast, aioseo, rankmath, wix, shopify, parked
AI crawler named inside a valid /llms.txt, e.g. GPTBot (see facet=bot for the values)
Minimum number of links in a valid /llms.txt, 0 or greater
true keeps only valid /llms.txt files that mention the Model Context Protocol
true keeps only valid /llms.txt files that mention OpenAPI or Swagger
true keeps only valid /llms.txt files that still hold template placeholder text
true keeps only valid /llms.txt files containing instruction-override text aimed at AI readers
true keeps only sites serving a valid /llms-full.txt
/robots.txt state. Enum: present, absent, blocked, error, skipped
Default stance of robots.txt toward AI crawlers. Enum: allowed, partial, blocked
Repeatable AI crawler that robots.txt must block (AND). Enum: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Applebot-Extended, Amazonbot, meta-externalagent, cohere-ai, Diffbot
Repeatable AI crawler; robots.txt must block at least one (OR). Enum: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Applebot-Extended, Amazonbot, meta-externalagent, cohere-ai, Diffbot
Repeatable AI crawler that robots.txt must NOT block. Enum: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Applebot-Extended, Amazonbot, meta-externalagent, cohere-ai, Diffbot
true keeps only sites whose robots.txt blocks all 17 tracked AI crawlers
Minimum number of AI training crawlers (of 11 tracked) that robots.txt blocks, 0 or greater
Content-Signal ai-train value declared in robots.txt. Enum: yes, no
Exact top-level-domain filter, e.g. com, org, io
Exact category label carried by the source list; the top-1M run has none, so this is empty for it
true keeps only sites whose homepage control request answered, false only sites that did not
true keeps only sites whose homepage was walled off from the scan (a block or challenge)
false (the common case) excludes backend CDN/DNS/cloud-vendor hostnames that rank highly but serve no public homepage; true keeps only those
Scan run id; defaults to the latest run
Headers
API key required
Response
OK
Example response
{
"code": 200,
"msg": "OK"
}Changes
No recorded changes to this operation across all 4 revisions of this API.