Sign in
Extract

Extract Async Endpoint

post/v1/extract/async

Request body

citystring

City for geolocation

consent_headerboolean

Whether to automatically handle cookie consent headers

device'desktop' | 'mobile' | 'tablet'

Device type for browser emulation

driver'vx6' | 'vx8' | 'vx8-pro' | 'vx10' | 'vx10-pro' | 'vx12' | 'vx12-pro' | 'media-vx6'

Browser driver to use

expected_status_codesinteger[]

Expected HTTP status codes for successful requests

formatsstring[]

List of acceptable response formats in order of preference

headersobject

Custom HTTP headers to include in the request

http2boolean

Whether to use HTTP/2 protocol

is_xhrboolean

Whether to emulate XMLHttpRequest behavior

markdown_backend'full_page' | 'main_content'

Selects which markdown conversion strategy to use. "full_page" converts the entire HTML page. "main_content" uses Mozilla Readability to extract the main article content before converting.

method'GET' | 'POST' | 'PUT' | 'PATCH' | 'DELETE'

HTTP method for the request

os'windows' | 'mac os' | 'linux' | 'android' | 'ios'

Operating system to emulate

parseboolean

Whether to parse the response content

renderboolean

Whether to render JavaScript content using a browser

request_timeoutnumber

Request timeout in milliseconds

state'AL' | 'AK' | 'AS' | 'AZ' | 'AR' | 'CA' | 'CO' | 'CT' | 'DE' | 'DC' | 'FL' | 'GA' | 'GU' | 'HI' | 'ID' | 'IL' | 'IN' | 'IA' | 'KS' | 'KY' | 'LA' | 'ME' | 'MD' | 'MA' | 'MI' | 'MN' | 'MS' | 'MO' | 'MT' | 'NE' | 'NV' | 'NH' | 'NJ' | 'NM' | 'NY' | 'NC' | 'ND' | 'MP' | 'OH' | 'OK' | 'OR' | 'PA' | 'PR' | 'RI' | 'SC' | 'SD' | 'TN' | 'TX' | 'UT' | 'VT' | 'VA' | 'VI' | 'WA' | 'WV' | 'WI' | 'WY'

US state for geolocation (only valid when country is US)

tagstring

User-defined tag for request identification

urlstring required

Target URL to scrape

callback_urlstring

URL to call back when async operation completes

storage_compressboolean

Whether to compress stored data

storage_object_namestring

Custom name for the stored object

storage_typestring

Type of storage to use for results

storage_urlstring

URL for storage location

Example request

{
  "browser": "chrome",
  "browser_actions": [
    {
      "goto": "https://example.com/login"
    },
    {
      "wait_for_element": "#login-form"
    },
    {
      "fill": {
        "selector": "#username",
        "value": "user@example.com"
      }
    },
    {
      "fill": {
        "selector": "#password",
        "value": "password123"
      }
    },
    {
      "click": "#submit"
    },
    {
      "screenshot": {
        "full_page": true
      }
    }
  ],
  "city": "Los Angeles",
  "cookies": "sessionId=abc123; userId=user456",
  "device": "desktop",
  "driver": "vx8",
  "expected_status_codes": [
    200,
    201
  ],
  "formats": [
    "html"
  ],
  "headers": {
    "Accept-Language": "en-US",
    "User-Agent": "CustomBot/1.0"
  },
  "http2": true,
  "is_xhr": true,
  "markdown_backend": "full_page",
  "method": "GET",
  "network_capture": [
    {
      "resource_type": "document"
    }
  ],
  "os": "windows",
  "parse": true,
  "render": true,
  "request_timeout": 30000,
  "skill": "dynamic-content",
  "state": "CA",
  "tag": "campaign-2024-q1",
  "callback_url": "https://example.com/webhook/callback",
  "storage_compress": true,
  "storage_object_name": "result-2024-01-15.json",
  "storage_type": "s3",
  "storage_url": "s3://bucket-name/path/to/object"
}

Response

Task created successfully

status'success' required

Status indicating the async task was created successfully.

Example response

{
  "task": {
    "batch_id": "4b0a90bf-c951-42e4-95b3-a95a65ba69fc",
    "created_at": "2024-01-15T10:30:00Z",
    "download_url": "https://api.webit.live/api/v2/tasks/123e4567-e89b-12d3-a456-426614174000/results",
    "error": "Connection timeout",
    "error_type": "timeout_error",
    "id": "123e4567-e89b-12d3-a456-426614174000",
    "modified_at": "2024-01-15T10:35:00Z",
    "state": "pending",
    "status_code": 200,
    "status_url": "https://api.webit.live/api/v2/tasks/123e4567-e89b-12d3-a456-426614174000"
  }
}

Changes