---
title: "Map a public site"
method: POST
path: "/web/map"
tags: ["Retrieval"]
---

# Map a public site

`POST /web/map`

## Request

- Base URL: `https://api.crawlora.net/api/v1`
- URL: `https://api.crawlora.net/api/v1/web/map`
- Auth: API key in header `x-api-key`

Maps a public site into a bounded, approximate URL list from its sitemaps and the seed page's links. The first request needs `url`; options are `scope`, `sitemap` (`include`, `only`, `skip`), `max_urls` (1 to 5000, default 1000) and `page_size` (1 to 500, default 100). Continue with `cursor`, optionally with `page_size`; the first page's snapshot lasts 30 minutes.

## Request body

- RetrievalplatformMapRequest
  - `cursor` string
  - `max_urls` integer
  - `page_size` integer
  - `scope` RetrievaltypesURLScope
    - `exclude_paths` string[]
    - `include_paths` string[]
    - `include_subdomains` boolean
    - `query_policy` 'preserve' | 'drop_tracking'
  - `sitemap` string
  - `url` string

## Response `200`

OK

- object: The Response struct is used for wrapping the API response with a code, message, and data.
  - `code` integer: Code is the HTTP status code or a custom code used to indicate the result of the request @example 200
  - `data` object: Data is the response data (can be any object or null) @example {"id": 1, "name": "John Doe"}
    - `coverage` string
    - `expires_at` string
    - `has_more` boolean
    - `items` WebretrievalDiscoveredURL[]
      - `discovered_via` string
      - `last_modified` string
      - `observed_at` string
      - `parent_url` string
      - `title` string
      - `url` string
    - `next_cursor` string
    - `partial` boolean
    - `truncated` boolean
    - `usage` RetrievaltypesUsage
      - `browser_renders` integer
      - `credits_charged` integer
      - `credits_released` integer
      - `credits_reserved` integer
      - `fetch_attempts` integer
      - `fetched_urls` integer
      - `input_tokens` integer
      - `output_tokens` integer
      - `search_requests` integer
      - `settlement_pending` boolean
    - `warnings` RetrievaltypesWarning[]
      - `code` 'source_failed' | 'source_not_indexed' | 'discovery_truncated' | 'ranking_window_limit' | 'imprecise_date_excluded' | 'content_unavailable' | 'cached_source_stale'
      - `message` string
      - `source_id` string
  - `msg` unknown

## Other responses

- `400`: Bad Request
- `401`: Unauthorized
- `402`: Payment Required
- `410`: Gone
- `429`: Too Many Requests
- `500`: Internal Server Error
- `503`: Service Unavailable

---

[API](https://skmtc.dev/crawlora/apis/crawlora-api.md) · [All operations](https://skmtc.dev/crawlora/apis/crawlora-api/llms.txt) · [OpenAPI document](https://skmtc.dev/crawlora/apis/crawlora-api/revisions/afac9b012a27?raw)