Web Extraction

Extract structured data

Changed on

Extract website data using your JSON Schema and instructions.

post/web/extract

Request

  • Base URL: https://api.context.dev/v1
  • URL: https://api.context.dev/v1/web/extract
  • Auth: HTTP bearer

Request body

urlstring uri required

The starting website URL to crawl and extract from. Must include http:// or https://.

schemaobject required

JSON Schema (not an example object) for the returned object. Image fields include relevant page image references.

instructionsstring

Optional extraction guidance, such as which facts to prioritize or how to interpret fields in the schema.

factCheckboolean

Require facts stated on the page. Unsupported fields become null or empty.

followSubdomainsboolean

When true, follow links on subdomains of the starting URL's domain.

maxPagesinteger

Maximum number of pages to analyze for extraction. Hard cap: 50. Defaults to 5.

maxDepthinteger

Optional maximum link depth from the starting URL (0 = only the starting page). If omitted, there is no crawl depth limit.

includeFramesboolean

When true, iframe contents are included in Markdown before extraction.

maxAgeMsinteger nullable

Return cached scrape results if a prior scrape for the same parameters is younger than this many milliseconds. Defaults to 7 days (604800000 ms).

waitForMsinteger nullable

Optional browser wait time in milliseconds after initial page load for each crawled page.

settleAnimationsboolean

Wait briefly for CSS animations and transitions to settle before reading each page.

stopAfterMsinteger

Soft time budget for the crawl in milliseconds. Min: 10000 (10s). Max: 110000 (110s). Defaults to 80000 (80s), or 110000 (110s) when browser actions are provided.

zdr'enabled' | 'disabled'

enabled turns on zero data retention. Returns 403 ZDR_NOT_ENABLED unless your organization has ZDR.

tagsstring[]

Labels for filtering usage in the dashboard.

Example request

{
  "schema": {
    "type": "object",
    "properties": {
      "mission_statement": {
        "type": "string",
        "description": "The company's stated mission."
      },
      "case_studies": {
        "type": "array",
        "items": {
          "type": "object",
          "properties": {
            "title": {
              "type": "string"
            },
            "url": {
              "type": "string"
            }
          },
          "required": [
            "title",
            "url"
          ],
          "additionalProperties": false
        }
      }
    },
    "required": [
      "mission_statement",
      "case_studies"
    ],
    "additionalProperties": false
  },
  "tags": [
    "production",
    "team-alpha"
  ]
}

Response

Successful response

partialboolean

True when the timeout ended processing and this response contains only usable results completed so far. Unfinished results are omitted.

statusstring required

Always ok on success.

urlstring required

The starting URL that was analyzed

urls_analyzedstring[] required

List of URLs whose Markdown was used for extraction

dataobject required

Extracted data matching the request schema

request_idstring uuid required

Unique ID of this request, also in X-Request-Id. Include it when contacting support.

Example response

{
  "request_id": "3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91"
}

Changes

  • 61627728926bRevision changes
    • warning

      added the new NO_PUBLISHED_MARKDOWN enum value to the response property for the response status

    • warning

      added the new NO_PUBLISHED_MARKDOWN enum value to the response property for the response status

    • warning

      added the new NO_PUBLISHED_MARKDOWN enum value to the response property for the response status

    • warning

      added the new NO_PUBLISHED_MARKDOWN enum value to the response property for the response status

    • warning

      added the new NO_PUBLISHED_MARKDOWN enum value to the response property for the response status

  • 4ddcf49b9fe5Revision changes
    • info

      added the non-success response with the status

    • info

      added the optional property to the response with the status

    • info

      added the optional property to the response with the status

    • info

      added the optional property to the response with the status

    • info

      added the optional property to the response with the status

    • info

      added the optional property to the response with the status