---
title: "Search the AI-readiness (llms.txt) dataset"
method: GET
path: "/datasets/llms/search"
tags: ["Datasets"]
---

# Search the AI-readiness (llms.txt) dataset

`GET /datasets/llms/search`

## Request

- Base URL: `https://api.crawlora.net/api/v1`
- URL: `https://api.crawlora.net/api/v1/datasets/llms/search`
- Auth: API key in header `x-api-key`

Searches the AI-readiness census of the top 1M sites (dataset id enum value `llms`) — one record per site describing whether it serves a real `/llms.txt` or `/llms-full.txt` (who produced it, what it links to and mentions) and what its `/robots.txt` says to 17 AI crawlers, next to a homepage control request. Search results omit the file bodies; read one site with the item endpoint. `blocked` and `error` states mean unknown, never absent. Sort enum: `relevance`, `rank_asc`, `links_desc`, `bytes_desc`, `domain_asc`, `crawled_desc`. state and robots_state enum: `present`, `absent`, `blocked`, `error`, `skipped`. source_kind enum: `shopify_agent_template`, `seo_plugin`, `platform_default`, `parked`, `custom`. generator enum: `yoast`, `aioseo`, `rankmath`, `wix`, `shopify`, `parked`. robots_default enum: `allowed`, `partial`, `blocked`. content_signal_train enum: `yes`, `no`. Crawler enum for blocks_all, blocks_any and not_blocked: `GPTBot`, `OAI-SearchBot`, `ChatGPT-User`, `ClaudeBot`, `Claude-SearchBot`, `Claude-User`, `anthropic-ai`, `Google-Extended`, `PerplexityBot`, `Perplexity-User`, `CCBot`, `Bytespider`, `Applebot-Extended`, `Amazonbot`, `meta-externalagent`, `cohere-ai`, `Diffbot`. Site owners can ask for removal at privacy@crawlora.net. Billing: this search costs 1 credit per record returned, with a minimum of 2 (an empty result costs 2, a full page of 100 costs 100).

## Query parameters

- `q` string
- `text` string
- `state` 'present' | 'absent' | 'blocked' | 'error' | 'skipped'
- `valid` boolean
- `source_kind` 'shopify_agent_template' | 'seo_plugin' | 'platform_default' | 'parked' | 'custom'
- `generator` 'yoast' | 'aioseo' | 'rankmath' | 'wix' | 'shopify' | 'parked'
- `bot` string
- `min_links` integer
- `mentions_mcp` boolean
- `mentions_openapi` boolean
- `placeholder` boolean
- `injection` boolean
- `full_valid` boolean
- `robots_state` 'present' | 'absent' | 'blocked' | 'error' | 'skipped'
- `robots_default` 'allowed' | 'partial' | 'blocked'
- `blocks_all` string[]
- `blocks_any` string[]
- `not_blocked` string[]
- `all_ai_blocked` boolean
- `min_train_blocked` integer
- `content_signal_train` 'yes' | 'no'
- `tld` string
- `category` string
- `reachable` boolean
- `home_blocked` boolean
- `is_infrastructure` boolean
- `run_id` string
- `sort` 'relevance' | 'rank_asc' | 'links_desc' | 'bytes_desc' | 'domain_asc' | 'crawled_desc'
- `page` integer
- `page_size` integer

## Headers

- `x-api-key` string, required

## Response `200`

OK

- DatasetsLlmsSearchResponseDoc
  - `code` integer
  - `data` DatasetsLlmsSearchResponse
    - `dataset` string
    - `items` DatasetsLlmsRecord[]
      - `analyzer_version` string
      - `category` string
      - `domain` string
      - `full` EsLlmsFile
        - `body` string: Body is the file content, capped by the worker (BodyTruncated says whether it was cut). Only present for a valid file.
        - `body_sha256` string
        - `body_truncated` boolean
        - `body_withheld` string: BodyWithheld is set only on the public API, never stored: it names why a stored body was not returned ("injection").
        - `bots` string[]
        - `bytes` integer
        - `content_type` string
        - `ctype_mismatch` boolean
        - `external_hosts` integer
        - `fetched_at` string
        - `final_url` string
        - `generator` string
        - `has_optional` boolean
        - `injection` boolean
        - `kind` string: Kind is the body classification (valid, html, robots, json, ...) for a 200.
        - `link_hosts` string[]
        - `links` integer
        - `links_with_note` integer
        - `llm_artifact` boolean
        - `markdown_links` integer
        - `mentions_api` boolean
        - `mentions_mcp` boolean
        - `mentions_openapi` boolean
        - `mentions_sitemap` boolean
        - `placeholder` boolean
        - `policy` string[]
        - `sections` integer
        - `source_kind` string
        - `state` string
        - `status` integer
        - `summary` string
        - `template_key` string
        - `title` string
        - `valid` boolean
      - `home_blocked` boolean
      - `home_challenge` boolean
      - `home_status` integer
      - `is_infrastructure` boolean
      - `llms` EsLlmsFile
        - `body` string: Body is the file content, capped by the worker (BodyTruncated says whether it was cut). Only present for a valid file.
        - `body_sha256` string
        - `body_truncated` boolean
        - `body_withheld` string: BodyWithheld is set only on the public API, never stored: it names why a stored body was not returned ("injection").
        - `bots` string[]
        - `bytes` integer
        - `content_type` string
        - `ctype_mismatch` boolean
        - `external_hosts` integer
        - `fetched_at` string
        - `final_url` string
        - `generator` string
        - `has_optional` boolean
        - `injection` boolean
        - `kind` string: Kind is the body classification (valid, html, robots, json, ...) for a 200.
        - `link_hosts` string[]
        - `links` integer
        - `links_with_note` integer
        - `llm_artifact` boolean
        - `markdown_links` integer
        - `mentions_api` boolean
        - `mentions_mcp` boolean
        - `mentions_openapi` boolean
        - `mentions_sitemap` boolean
        - `placeholder` boolean
        - `policy` string[]
        - `sections` integer
        - `source_kind` string
        - `state` string
        - `status` integer
        - `summary` string
        - `template_key` string
        - `title` string
        - `valid` boolean
      - `method_version` string
      - `origin` string
      - `probe_error` string
      - `probed_at` string
      - `rank` integer
      - `reachable` boolean
      - `robots` EsLlmsRobots
        - `all_ai_blocked` boolean
        - `bytes` integer
        - `content_signal` object
        - `crawl_delay_secs` integer
        - `crawlers` object
        - `default_stance` string
        - `fetched_at` string
        - `groups` integer
        - `kind` string
        - `mentions_ai` boolean
        - `mentions_llms_txt` boolean
        - `rules` integer
        - `sitemaps` string[]
        - `state` string
        - `status` integer
        - `train_blocked` integer
      - `run_date` string
      - `run_id` string
      - `seed_source` string
      - `tld` string
    - `page` integer
    - `page_size` integer
    - `sort` string
    - `total` integer
  - `msg` string

## Other responses

- `400`: Bad Request
- `429`: Too Many Requests
- `500`: Internal Server Error

---

[API](https://skmtc.dev/crawlora/apis/crawlora-api.md) · [All operations](https://skmtc.dev/crawlora/apis/crawlora-api/llms.txt) · [OpenAPI document](https://skmtc.dev/crawlora/apis/crawlora-api/revisions/afac9b012a27?raw)