Datasets

Get a site from the AI-readiness (llms.txt) dataset

Returns one site by its domain from dataset id enum value llms, including the stored /llms.txt and /llms-full.txt bodies (llms.body, full.body; body_truncated marks a cut file, and a body that tries to override an AI reader's instructions is withheld with body_withheld set to injection), their structure and classification, and the /robots.txt stance toward each tracked AI crawler. Returns 404 when the domain is not in the dataset or its owner asked for removal (privacy@crawlora.net).

get/datasets/llms/items/{domain}

Request

  • Base URL: https://api.crawlora.net/api/v1
  • URL: https://api.crawlora.net/api/v1/datasets/llms/items/{domain}
  • Auth: API key in header x-api-key

Path parameters

domainstring required

Site domain, e.g. shopify.com (a scheme and www. are stripped)

Headers

x-api-keystring required

API key required

Response

OK

codeinteger
msgstring

Example response

{
  "code": 200,
  "msg": "OK"
}

Changes

No recorded changes to this operation across all 2 revisions of this API.