---
title: "Robots.txt Parse (x402)"
method: GET
path: "/x402/v2/robots-txt-parse"
tags: ["x402"]
---

# Robots.txt Parse (x402)

`GET /x402/v2/robots-txt-parse`

## Request

- Base URL: `https://api.strale.io`
- URL: `https://api.strale.io/x402/v2/robots-txt-parse`
- Auth: none required

Fetch and parse a website's robots.txt. Returns rules per user-agent, sitemaps, crawl delays, and whether all crawlers are blocked.

## Query parameters

- `url` string, required

## Response `200`

Success

- object
  - `rules` unknown[]
    - unknown
  - `exists` boolean
  - `sitemaps` unknown[]
    - unknown
  - `user_agents` unknown[]
    - unknown
  - `blocks_all_crawlers` boolean

## Other responses

- `402`: Payment required (x402 — pay with USDC on Base)

---

[API](https://skmtc.dev/strale/apis/strale-api-2.md) · [All operations](https://skmtc.dev/strale/apis/strale-api-2/llms.txt) · [OpenAPI document](https://skmtc.dev/strale/apis/strale-api-2/revisions/a5293878cfc6?raw)