---
title: "Generate llms.txt (and optionally llms-full.txt) from a real analysis of the site: fetches the target page, robots.txt, and sitemap.xml, then reads up to 8 key same-domain pages (about/docs/pricing/blog/api/contact/faq and similar) for their titles, descriptions, and main text."
method: POST
path: "/tools/generate_llms_txt"
tags: ["tools"]
---

# Generate llms.txt (and optionally llms-full.txt) from a real analysis of the site: fetches the target page, robots.txt, and sitemap.xml, then reads up to 8 key same-domain pages (about/docs/pricing/blog/api/contact/faq and similar) for their titles, descriptions, and main text.

`POST /tools/generate_llms_txt`

## Request

- Base URL: `https://www.crawlforge.dev/api/v1`
- URL: `https://www.crawlforge.dev/api/v1/tools/generate_llms_txt`
- Auth: API key in header `X-API-Key`

## Request body

- GenerateLlmsTxtRequest
  - `url` string, uri, required: The website URL to generate llms.txt for
  - `respect_robots` boolean: Fetch each origin's robots.txt and skip URLs it disallows for CrawlForge. A missing or unreachable robots.txt is treated as no restrictions. Returns 403 ROBOTS_DISALLOWED when the target URL itself is disallowed, with no credits charged; disallowed pages are left out of the analysis and counted in warnings. analysisOptions.respectRobots is an alias — either one set to false disables the gate.
  - `format` 'both' | 'llms-txt' | 'llms-full-txt': Output format; llms-full.txt inlines up to 2000 characters of text per analyzed page
  - `complianceLevel` 'basic' | 'standard' | 'strict': Accepted for compatibility; content is derived from real site data, not canned policy text
  - `analysisOptions` object: Analysis options. maxPages is clamped to 30 and maxDepth to 2 on the hosted REST API (clamps are reported in clamps_applied). respectRobots skips pages disallowed by robots.txt; detectAPIs flags /api, /docs/api, openapi, and swagger links found in fetched HTML.
    - `maxDepth` number
    - `maxPages` number
    - `respectRobots` boolean
    - `detectAPIs` boolean
    - `analyzeContent` boolean
    - `checkSecurity` boolean
  - `outputOptions` object: Output customization: organizationName (overrides the site title), contactEmail, customGuidelines, customRestrictions
    - `organizationName` string
    - `contactEmail` string, email
    - `customGuidelines` string[]
    - `customRestrictions` string[]
    - `includeDetailed` boolean
    - `includeAnalysis` boolean
  - `max_inline_chars` integer: Largest result returned inline, in characters of its JSON (1000-10000000; env CRAWLFORGE_MAX_INLINE_CHARS sets the default). Over it, the result is stored for 1 hour and the response carries a preview, a result_handle, total_chars and truncated: true; read the rest with read_result (1 credit).

## Response `200`

The tool's result; 5 credits charged (see x-credits-note and x-extra for add-ons).

- ToolSuccess: The envelope every tool returns on success. Credits are deducted only after the tool succeeded, so credits_used is the price of this call and an error response never charges.
  - `success` true, required
  - `data` object, required: The tool's result. Its shape is tool-specific and documented on the tool's docs page (x-docs-url on the operation).
  - `credits_used` integer, required: Credits charged for this call: the operation's x-credits, plus any add-on x-credits-note or x-extra describes.
  - `credits_remaining` integer, required: The account's credit balance after this call.
  - `processing_time` integer, required: Server-side processing time in milliseconds.
  - `warnings` string[]: Non-fatal notes: a robots.txt override that was recorded, a format that could not be produced, a redaction that changed the payload.

## Other responses

- `400`: VALIDATION_ERROR: the body does not match the request schema.
- `401`: MISSING_API_KEY or INVALID_API_KEY.
- `402`: INSUFFICIENT_CREDITS: the balance does not cover the tool's base price. X-Auto-Recharge-Triggered says whether an automatic recharge was started; retry once it lands.
- `429`: RATE_LIMIT_EXCEEDED: your plan's rate limit. Retry-After: 60.
- `500`: TOOL_ERROR, INTERNAL_ERROR or CREDIT_DEDUCTION_FAILED.
- `default`: Any other error. Which of 403, 404, 413, 422, 501, 502, 503 and 504 a tool can return depends on what it does (fetching a page, calling the execution backend, reading stored results); the code vocabulary is listed on the ErrorResponse schema.

---

[API](https://skmtc.dev/crawlforge/apis/crawlforge-rest-api.md) · [All operations](https://skmtc.dev/crawlforge/apis/crawlforge-rest-api/llms.txt) · [OpenAPI document](https://skmtc.dev/crawlforge/apis/crawlforge-rest-api/revisions/a5532fa8d5d5?raw)