Datasets

Search the AI-readiness (llms.txt) dataset

Searches the AI-readiness census of the top 1M sites (dataset id enum value llms) — one record per site describing whether it serves a real /llms.txt or /llms-full.txt (who produced it, what it links to and mentions) and what its /robots.txt says to 17 AI crawlers, next to a homepage control request. Search results omit the file bodies; read one site with the item endpoint. blocked and error states mean unknown, never absent. Sort enum: relevance, rank_asc, links_desc, bytes_desc, domain_asc, crawled_desc. state and robots_state enum: present, absent, blocked, error, skipped. source_kind enum: shopify_agent_template, seo_plugin, platform_default, parked, custom. generator enum: yoast, aioseo, rankmath, wix, shopify, parked. robots_default enum: allowed, partial, blocked. content_signal_train enum: yes, no. Crawler enum for blocks_all, blocks_any and not_blocked: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Applebot-Extended, Amazonbot, meta-externalagent, cohere-ai, Diffbot. Site owners can ask for removal at privacy@crawlora.net. Billing: this search costs 1 credit per record returned, with a minimum of 2 (an empty result costs 2, a full page of 100 costs 100).

get/datasets/llms/search

Request

  • Base URL: https://api.crawlora.net/api/v1
  • URL: https://api.crawlora.net/api/v1/datasets/llms/search
  • Auth: API key in header x-api-key

Query parameters

qstring

Substring match on the site domain, max 256 characters

textstring

Full-text match on the /llms.txt title and summary, max 256 characters

state'present' | 'absent' | 'blocked' | 'error' | 'skipped'

/llms.txt state. Enum: present, absent, blocked, error, skipped. blocked and error mean unknown, not absent

validboolean

true keeps only sites serving a valid /llms.txt, false keeps the rest

source_kind'shopify_agent_template' | 'seo_plugin' | 'platform_default' | 'parked' | 'custom'

Producer of a valid /llms.txt. Enum: shopify_agent_template, seo_plugin, platform_default, parked, custom

generator'yoast' | 'aioseo' | 'rankmath' | 'wix' | 'shopify' | 'parked'

Generator detected in a valid /llms.txt. Enum: yoast, aioseo, rankmath, wix, shopify, parked

botstring

AI crawler named inside a valid /llms.txt, e.g. GPTBot (see facet=bot for the values)

min_linksinteger

Minimum number of links in a valid /llms.txt, 0 or greater

mentions_mcpboolean

true keeps only valid /llms.txt files that mention the Model Context Protocol

mentions_openapiboolean

true keeps only valid /llms.txt files that mention OpenAPI or Swagger

placeholderboolean

true keeps only valid /llms.txt files that still hold template placeholder text

injectionboolean

true keeps only valid /llms.txt files containing instruction-override text aimed at AI readers

full_validboolean

true keeps only sites serving a valid /llms-full.txt

robots_state'present' | 'absent' | 'blocked' | 'error' | 'skipped'

/robots.txt state. Enum: present, absent, blocked, error, skipped

robots_default'allowed' | 'partial' | 'blocked'

Default stance of robots.txt toward AI crawlers. Enum: allowed, partial, blocked

blocks_allstring[]

Repeatable AI crawler that robots.txt must block (AND). Enum: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Applebot-Extended, Amazonbot, meta-externalagent, cohere-ai, Diffbot

blocks_anystring[]

Repeatable AI crawler; robots.txt must block at least one (OR). Enum: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Applebot-Extended, Amazonbot, meta-externalagent, cohere-ai, Diffbot

not_blockedstring[]

Repeatable AI crawler that robots.txt must NOT block. Enum: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Applebot-Extended, Amazonbot, meta-externalagent, cohere-ai, Diffbot

all_ai_blockedboolean

true keeps only sites whose robots.txt blocks all 17 tracked AI crawlers

min_train_blockedinteger

Minimum number of AI training crawlers (of 11 tracked) that robots.txt blocks, 0 or greater

content_signal_train'yes' | 'no'

Content-Signal ai-train value declared in robots.txt. Enum: yes, no

tldstring

Exact top-level-domain filter, e.g. com, org, io

categorystring

Exact category label carried by the source list; the top-1M run has none, so this is empty for it

reachableboolean

true keeps only sites whose homepage control request answered, false only sites that did not

home_blockedboolean

true keeps only sites whose homepage was walled off from the scan (a block or challenge)

is_infrastructureboolean

false (the common case) excludes backend CDN/DNS/cloud-vendor hostnames that rank highly but serve no public homepage; true keeps only those

run_idstring

Scan run id; defaults to the latest run

sort'relevance' | 'rank_asc' | 'links_desc' | 'bytes_desc' | 'domain_asc' | 'crawled_desc'

Sort enum: relevance, rank_asc, links_desc, bytes_desc, domain_asc, crawled_desc

pageinteger

Page number, defaults to 1

page_sizeinteger

Page size, defaults to 20 and maxes at 100; page * page_size must be <= 10000

Headers

x-api-keystring required

API key required

Response

OK

codeinteger
msgstring

Example response

{
  "code": 200,
  "msg": "OK"
}

Changes

No recorded changes to this operation across all 3 revisions of this API.