COcontext
Web Extraction

Extract Structured Website Data

Changed on

Crawl a website, use the provided JSON Schema and instructions to prioritize relevant internal links, and extract structured data from the selected pages.

post/web/extract

Request

  • Base URL
    https://api.context.dev/v1
  • URL
    https://api.context.dev/v1/web/extract
  • Auth
    HTTP bearer

Request body

urlstring uri required

The starting website URL to crawl and extract from. Must include http:// or https://.

schemaobject required

JSON Schema for the returned data object. Image fields such as image_urls or product_photos automatically make page image references available to extraction, so product data and photos can be returned in one call. TypeScript Zod users can pass a JSON Schema generated from a Zod object; Python users can pass the equivalent JSON Schema object.

instructionsstring

Optional extraction guidance, such as which facts to prioritize or how to interpret fields in the schema.

factCheckboolean

When true, every returned value must be grounded in facts stated on the page; fields that cannot be supported by the page are returned as null/empty. When false (default), the model may make reasonable inferences and derivations from the page content (e.g. ideal customer, competitor analysis, recommendations) while keeping verifiable specifics (names, quotes, URLs, dates, metrics) faithful to the source.

followSubdomainsboolean

When true, follow links on subdomains of the starting URL's domain.

maxPagesinteger

Maximum number of pages to analyze for extraction. Hard cap: 50. Defaults to 5.

maxDepthinteger

Optional maximum link depth from the starting URL (0 = only the starting page). If omitted, there is no crawl depth limit.

includeFramesboolean

When true, iframe contents are included in Markdown before extraction.

maxAgeMsinteger

Return cached scrape results if a prior scrape for the same parameters is younger than this many milliseconds. Defaults to 7 days (604800000 ms).

waitForMsinteger

Optional browser wait time in milliseconds after initial page load for each crawled page.

settleAnimationsboolean

When true, waits briefly for CSS and transition animations to settle before extracting each crawled page. Defaults to false. This adds a bit of latency in exchange for more stable output on animated pages.

stopAfterMsinteger

Soft time budget for the crawl in milliseconds. Min: 10000 (10s). Max: 110000 (110s). Defaults to 80000 (80s), or 110000 (110s) when browser actions are provided.

timeoutMSinteger

Optional timeout in milliseconds for the request. If the request takes longer than this value, it will be aborted with a 408 status code. Maximum allowed value is 300000ms (5 minutes).

tagsstring[]

Optional tags for tracking usage. Up to 20 tags, each 1 to 50 characters.

Example request

{
  "schema": {
    "type": "object",
    "properties": {
      "mission_statement": {
        "type": "string",
        "description": "The company's stated mission."
      },
      "case_studies": {
        "type": "array",
        "items": {
          "type": "object",
          "properties": {
            "title": {
              "type": "string"
            },
            "url": {
              "type": "string"
            }
          },
          "required": [
            "title",
            "url"
          ],
          "additionalProperties": false
        }
      }
    },
    "required": [
      "mission_statement",
      "case_studies"
    ],
    "additionalProperties": false
  },
  "tags": [
    "production",
    "team-alpha"
  ]
}

Response

Successful response

statusstring required

Status of the response, e.g., 'ok'

urlstring required

The starting URL that was analyzed

urls_analyzedstring[] required

List of URLs whose Markdown was used for extraction

dataobject required

Extracted data matching the request schema

Changes

  • 1febe5476266Revision changes
    • info

      added the required property to the response with the status

  • 3e8f4b6acab4Revision changes
    • info

      added scroll discriminator mapping keys to the / request property

    • info

      added to the / request property oneOf list