---
title: "robots.txt parser"
method: GET
path: "/robots-txt"
tags: ["Domain"]
---

# robots.txt parser

`GET /robots-txt`

Fetches and parses the robots.txt file for any domain. Returns crawl rules, sitemap URLs, and per-agent directives.

## Query parameters

- `url` string, required

## Response `200`

Parsed robots.txt

- object
  - `url` string
  - `sitemaps` string[]
  - `agents` object

## Other responses

- `400` — Invalid request parameters
- `401` — Missing or invalid API key
- `429` — Rate limit exceeded

---

[API](https://skmtc.dev/kiprio/apis/kiprio-api.md) · [All operations](https://skmtc.dev/kiprio/apis/kiprio-api/llms.txt) · [OpenAPI document](https://skmtc-service-production.skmtc.workers.dev/v1/apis/kiprio/kiprio-api/revisions/bbe8f8fb1a35/schema)