---
title: "Build robots.txt that treats search engines, AI answer bots and AI training crawlers separately; validate llms.txt."
method: POST
path: "/api/v1/robots-txt"
tags: ["Web Tools"]
---

# Build robots.txt that treats search engines, AI answer bots and AI training crawlers separately; validate llms.txt.

`POST /api/v1/robots-txt`

## Request

- Base URL: `https://programmes.net`
- URL: `https://programmes.net/api/v1/robots-txt`
- Auth: none declared

robots.txt Builder & llms.txt Validator. Operations: `build` — Build robots.txt: allow or block search engines, AI answer engines/assistants and AI training crawlers (named), with Content-Signal and Sitemap lines. `validate-llms` — Check an llms.txt file against llmstxt.org: one H1, a > summary, H2 sections of Markdown link lists.

## Request body

- union
  - RobotsTxtBuild: Build robots.txt: allow or block search engines, AI answer engines/assistants and AI training crawlers (named), with Content-Signal and Sitemap lines.
    - `operation` 'build', required
    - `input` string: One item per line: sitemap URLs (https://…) and paths every crawler should skip (/admin/). May be empty.
    - `options` object
      - `search` 'allow' | 'block': Search engines and every crawler not named (the * group).
      - `aiAnswer` 'allow' | 'block': AI answer engines and assistants that fetch pages to answer a question.
      - `aiTrain` 'allow' | 'block': Crawlers that collect text to train AI models.
      - `contentSignal` boolean: Add contentsignals.org Content-Signal lines matching the choices.
  - RobotsTxtValidateLlms: Check an llms.txt file against llmstxt.org: one H1, a > summary, H2 sections of Markdown link lists.
    - `operation` 'validate-llms', required
    - `input` string: The llms.txt file to check.
    - `options` object

## Response `200`

The operation ran.

- object
  - `ok` unknown, required
  - `tool` 'robots-txt', required
  - `operation` 'build' | 'validate-llms', required
  - `result` unknown, required

## Other responses

- `400`: The request is not valid (https://programmes.net/api#invalid-request); The tool has no such operation (https://programmes.net/api#unknown-operation)
- `404`: No such tool (https://programmes.net/api#unknown-tool)
- `413`: The request or its input is too large (https://programmes.net/api#content-too-large)
- `415`: The body must be JSON (https://programmes.net/api#unsupported-media-type)
- `422`: An option is missing, of the wrong type or out of range (https://programmes.net/api#invalid-options); This call needs more CPU than one API call may use (https://programmes.net/api#refused); The operation could not process the input (https://programmes.net/api#tool-error)
- `429`: Too many requests (https://programmes.net/api#rate-limited)
- `500`: Internal error (https://programmes.net/api#internal-error)

---

[API](https://skmtc.dev/programmes/apis/programmes-net-api.md) · [All operations](https://skmtc.dev/programmes/apis/programmes-net-api/llms.txt) · [OpenAPI document](https://skmtc.dev/programmes/apis/programmes-net-api/revisions/aa939784a2e6?raw)