---
title: "Map URLs"
method: GET
path: "/web/urls"
tags: ["Web Scraping"]
---

# Map URLs

`GET /web/urls`

## Request

- Base URL: `https://api.context.dev/v1`
- URL: `https://api.context.dev/v1/web/urls`
- Auth: HTTP bearer

Discover a site's URLs, with page titles, descriptions, keywords, and language when available. Metadata can be missing on newly discovered URLs.

## Query parameters

- `domain` string, required: Domain to map, e.g. `stripe.com`.
- `includeSubdomains` boolean: Include URLs on subdomains.
- `maxLinks` integer: Maximum number of URLs to return.
- `sitemapUrl` string, uri: Fetch this sitemap instead of discovering sitemaps. Must belong to the domain or a subdomain.
- `urlRegex` string: Regex pattern. Only URLs matching this pattern are returned and counted against maxLinks.
- `excludeUrlRegex` string: Regex pattern. URLs matching this pattern are not returned, even when they match urlRegex.
- `search` string: Filter URLs by a topic or phrase, most relevant first.
- `headers` object: HTTP headers for the target origin. Non-empty headers bypass caching.
- `timeoutOpts` LenientPartialTimeout: Request deadline and what to return when it passes.
  - `milliseconds` integer, required: Deadline in milliseconds.
  - `behavior` 'fail' | 'return-partial': "fail" returns 408 at the deadline. "return-partial" returns available results; inspect the response’s partial flag.
- `zdr` 'enabled' | 'disabled': `enabled` turns on zero data retention. Returns 403 `ZDR_NOT_ENABLED` unless your organization has ZDR.
- `tags` string[]: Labels for filtering usage in the dashboard.

## Response `200`

Discovered URLs with available page metadata

- object
  - `success` true, required
  - `domain` string, required
  - `urls` object[], required
    - `url` string, required
    - `title` string
    - `description` string
    - `keywords` string[]
    - `language` string
  - `partial` boolean
  - `key_metadata` KeyMetadata: Credits this request used and your remaining balance.
    - `credits_consumed` integer, required: Credits charged for this request.
    - `credits_remaining` integer, required: Credits remaining for your organization.
  - `request_id` string, uuid, required: Unique ID of this request, also in `X-Request-Id`. Include it when contacting support.

## Other responses

- `400`: Bad request
- `401`: Missing, invalid, or disabled API key.
- `402`: Insufficient credits (`USAGE_EXCEEDED`); authentication is valid. Retry once funded. Legacy deployments may return 401.
- `403`: API key permissions or organization settings do not allow this request.
- `408`: Request timeout
- `429`: Rate limit exceeded. Organization concurrency rejections include structured error_data and subscription settings; follow Retry-After before retrying.
- `500`: Internal server error

## Changes

- **2026-10-21** `61627728926b`: 4 warning
  - added the new `NO_PUBLISHED_MARKDOWN` enum value to the `error_code` response property for the response status `400`
  - added the new `NO_PUBLISHED_MARKDOWN` enum value to the `error_code` response property for the response status `401`
  - added the new `NO_PUBLISHED_MARKDOWN` enum value to the `error_code` response property for the response status `402`
  - added the new `NO_PUBLISHED_MARKDOWN` enum value to the `error_code` response property for the response status `403`
- **2026-10-19** `aeb64a2b7925`: 1 info
  - added the new optional `query` request parameter `excludeUrlRegex`
- **2026-10-19** `4ddcf49b9fe5`: 5 info
  - added the non-success response with the status `402`
  - added the optional property `error_data` to the response with the `400` status
  - added the optional property `error_data` to the response with the `401` status
  - added the optional property `error_data` to the response with the `403` status
  - …1 more
- **2026-10-16** `2125f3e94e52`: 1 info
  - endpoint added

[Operation history](https://skmtc.dev/context/apis/brand-api/changes/web/urls/get.md)

---

[API](https://skmtc.dev/context/apis/brand-api.md) · [All operations](https://skmtc.dev/context/apis/brand-api/llms.txt) · [OpenAPI document](https://skmtc.dev/context/apis/brand-api/revisions/9e2ab7841fb1?raw)