clusters

Extract values from the sites in a cluster

Pulls values out of the page source of the sites in the cluster, part by part: domains [offset, offset+limit). Call again with offset = next_offset until it is null. Costs extraction points: a site found in the index costs as many points as values found, 0.1 if none; when the day's points run out the call stops early (stopped: extract_quota_exceeded). format=csv gives one row per site with X-Next-Offset and X-Points-Left headers.

get/v1/clusters/{id}/extract

Request

  • Base URL: https://api.publicwww.com
  • URL: https://api.publicwww.com/v1/clusters/{id}/extract
  • Auth: HTTP bearer

Path parameters

idinteger required

Cluster id, as in the list.

Query parameters

presetsstring

Ready-made expressions by name, comma separated: email,phone. See /v1/clusters/presets.

regexstring

Your own expression: PCRE in slashes (or pipes) with optional i, m, s, u flags; the first capture group is the value. Several: newline separated, or an array in a POST body. Up to 10 in all, presets included.

offsetinteger

Where to start in the cluster: next_offset from the previous call.

limitinteger

Domains to go through in this call, up to 1000.

skip_emptyboolean

Leave out sites where nothing matched (default 1).

format'json' | 'xml' | 'csv'

json or xml or csv.

Response

Extracted values.

clusterinteger required
namestring
sizeinteger
offsetinteger required
scannedinteger required

Domains gone through in this call.

in_indexinteger
with_matchesinteger
next_offsetinteger nullable required

Pass as offset to continue; null when the cluster is done.

regexstring[] required

Expressions applied, presets resolved, one column each.

points_usednumber required
points_leftinteger required
stopped'extract_quota_exceeded' | 'time_limit'

Changes

No recorded changes to this operation across all 1 revision of this API.