web

Map URLs on a site

Discovers URLs for a public site and returns them as a normalized WebPageList.

get/web/map

Request

  • Base URL: https://www.socialcrawl.dev/v1
  • URL: https://www.socialcrawl.dev/v1/web/map
  • Auth: API key in header x-api-key

Query parameters

urlstring required

Site URL to map.

searchstring

Optional path or keyword filter.

limitinteger

Maximum URLs to return, up to 5000.

sitemap'include' | 'skip' | 'only'

How to use the site's sitemap.xml: 'include' (default) to merge it with crawled links, 'only' to trust it alone, 'skip' to ignore it.

include_subdomainsboolean

Follow links onto subdomains of the root URL. Default true.

ignore_query_parametersboolean

Treat URLs differing only by query string as one URL, which keeps paginated and tracking-tagged duplicates out of the result. Default true.

freshboolean

Bypass the upstream cache and re-discover the site now. Slower; use it when the map is known to be stale.

goalstring

Optional. What you are looking for on the site, in your own words. The map still returns every URL, ranked, with skipped_urls listing the ones that look unlikely to hold it (each with p). Without this param the map is unchanged.

Headers

Cache-Controlstring

Send no-cache to bypass the response cache and force a live fetch. Billed at the normal endpoint cost; the fresh result is written back to cache for the next caller. Only the no-cache directive triggers this. See the Response Schema guide for details.

Idempotency-Keystring

Optional UUID that makes the request safely retriable. The key is reserved atomically per account before billing. A concurrent retry receives 409 IDEMPOTENCY_IN_PROGRESS with a Retry-After of 1 second while the original is young, growing to a quarter of the time it has been running (at most 30); a crashed reservation expires after 5 minutes. A replay keeps the cached payload immutable except for billing metadata: credits_used becomes 0, idempotent_replay becomes true, and credits_remaining is refreshed to the current balance. A known current balance appears in both the body and X-Credits-Remaining header; no balance row resolves to 0. On a transient lookup failure, body credits_remaining is null and X-Credits-Remaining is omitted. Completed outcomes expire after 24 hours. If the claim store is unavailable, the request fails closed before billing. GET endpoints do not combine idempotent replay with Server-Sent Events: use Accept: application/json when sending this header. The unsupported key-plus-SSE combination is rejected before billing.

Response

Successful response

successboolean required

Whether the request succeeded

platformstring required

Platform name

endpointstring required

API endpoint path

credits_usedinteger required

Number of credits consumed

credits_remaininginteger nullable required

Current account balance, or null when this response could not establish one. Free cache hits always return null because they do not establish an authoritative post-request balance. An idempotency replay also returns null when its current-balance lookup fails. It is never a fabricated zero, so a zero here means the account really is empty. The authoritative reading is the one on a billed response, which comes straight from the deduction.

request_idstring required

Unique request identifier for support

cachedboolean required

Whether the response was served from cache

idempotent_replayboolean

True only when this response is an idempotency replay

Example response

{
  "success": true,
  "platform": "web",
  "endpoint": "/v1/web/map",
  "credits_used": 1,
  "credits_remaining": 4999,
  "request_id": "req-abc123",
  "idempotent_replay": true,
  "meta": {
    "hint": {
      "code": "search_multi",
      "message": "You searched \"matcha\" on 3 platforms one call at a time; /v1/search/multi returns each platform's own search rows in one call at the same price per platform, and a platform with no rows is not charged.",
      "method": "GET",
      "path": "/v1/search/multi?query=matcha&platforms=instagram,tiktok,youtube"
    }
  },
  "pagination": {
    "next_cursor": "sc.eyJ2IjoyLCJjIjoiMiIsInAiOiJwYWdlIn0",
    "has_more": true,
    "page_size": 30,
    "stopped_at": "known_id"
  }
}

Changes

No recorded changes to this operation across all 1 revision of this API.