--- title: "List async web jobs" method: GET path: "/web/jobs" tags: ["web"] --- # List async web jobs `GET /web/jobs` ## Request - Base URL: `https://www.socialcrawl.dev/v1` - URL: `https://www.socialcrawl.dev/v1/web/jobs` - Auth: API key in header `x-api-key` Lists async crawl, batch scrape, and agent jobs for the API key. ## Query parameters - `limit` integer - `cursor` string ## Headers - `Cache-Control` string - `Idempotency-Key` string ## Response `200` Successful response - object - `success` boolean, required: Whether the request succeeded - `platform` string, required: Platform name - `endpoint` string, required: API endpoint path - `data` object, required: Platform-specific response data - `items` object[]: Array of canonical web page wrappers ({ page }) - `page` object: Canonical web page object with identity, content, media, extraction, and fetch provenance fields. - `url` string, nullable: Requested URL. - `final_url` string, nullable: Final resolved URL after redirects. - `status_code` number, nullable: HTTP status code observed while fetching the page. - `scrape_id` string, nullable: Opaque scrape identifier for follow-up interactions. - `fetched_at` string, nullable: Fetch timestamp when reported. - `published_at` string, nullable: When the page was published: an ISO 8601 instant, or a YYYY-MM-DD date when only the day is printed. From the page's own article:published_time meta tag on a scrape, and from the news date or that tag on a search row. Null when the page states none. - `content` object: Extracted page content. - `markdown` string, nullable: Markdown body. - `html` string, nullable: HTML body. - `raw_html` string, nullable: Raw HTML body. - `summary` string, nullable: Generated summary. - `media` object: Short-lived media URLs returned for the page. - `screenshot_url` string, nullable: Screenshot URL when requested. - `audio_url` string, nullable: Audio URL. - `video_url` string, nullable: Video URL. - `extraction` object, nullable: Structured extraction result when requested. On `web/extract` every value is checked against the page text: a placeholder such as 'Not specified', or a value the page does not state, comes back as null and is listed in `extraction_check.removed`. - `extraction_check` object: `web/extract` only. How each extracted value was checked against the page text. A value is null in `extraction` when it was a placeholder or the page does not state it; `removed` lists those fields with the value the extractor gave, so nothing is hidden. A value kept without a check (the check was unavailable, or the page is too long to check whole) is named in `unverified` and in `_warnings`. - `checked` integer: Non-null values examined. - `found_in_text` integer: Values kept because they appear word for word on the page. - `confirmed` integer: Values kept because a check found the page states them, for example in another language. - `removed` object[]: Fields set to null, with the value the extractor gave. - `field` string: Path of the field, e.g. `faq[2].answer`. - `value` unknown - `reason` 'placeholder' | 'not_on_page': `placeholder`: the value holds no answer, either a marker such as 'Not specified' or '/', or a bare link path such as `/sub/info.php` in a field that does not ask for a link. `not_on_page`: the page does not state it. - `unverified` string[]: Fields returned as extracted without a check. - `fetch` object: Vendor-neutral fetch provenance. - `cache_state` string, nullable: Upstream cache state when reported. - `cached_at` string, nullable: Upstream cache timestamp when reported. - `proxy_tier` string, nullable: Proxy tier used for the fetch. - `next_cursor` string, nullable: Opaque cursor for the next page. Pass it back as a query parameter on endpoints that support pagination. Present only when the upstream reports more results. - `total` integer, nullable: Total number of matching results, when the upstream provides a count. Omitted otherwise. - `dropped` integer, required: Number of upstream list items dropped because they could not be repaired to the endpoint schema. Valid list responses include 0. - `_warnings` string[]: Non-fatal notices about this response (field-map drift, clamped computed values). Advisory only: its presence never means the request failed. Omitted entirely when there is nothing to report, so treat absent as 'no warnings'. - `credits_used` integer, required: Number of credits consumed - `credits_remaining` integer, nullable, required: Current account balance, or null when this response could not establish one. Free cache hits always return null because they do not establish an authoritative post-request balance. An idempotency replay also returns null when its current-balance lookup fails. It is never a fabricated zero, so a zero here means the account really is empty. The authoritative reading is the one on a billed response, which comes straight from the deduction. - `request_id` string, required: Unique request identifier for support - `cached` boolean, required: Whether the response was served from cache - `idempotent_replay` boolean: True only when this response is an idempotency replay - `meta` object: Optional. Present only when this call completed a pattern that one other call does more cheaply or in one step, at most once per pattern per API key per day. Send the header `X-SocialCrawl-Hints: off` to never receive it. - `hint` object: One suggestion: what to call instead and why, with the price when the suggested call costs more than the calls it replaces. - `code` string: creator_card | batch_profiles | search_multi | incremental_sync | comment_labels | search_repeat - `message` string: One sentence for a person or an agent to read. - `method` string: GET or POST. - `path` string: The call the hint points to. - `pagination` object, required: Cursor state for this page. Present on every list response. - `next_cursor` string, nullable, required: Opaque token to send back as `cursor` for the next page, or null at end-of-list. Pass it back verbatim; never decode or trim it. - `has_more` boolean, required: Explicit stop signal. Prefer this over inspecting next_cursor or comparing against total. - `page_size` integer, required: Number of items in THIS page. - `stopped_at` 'since' | 'known_id' | 'end' | 'null', nullable: Present only when the request sent `since` or `stop_at_id`. Why the walk stopped: `since` (this page reached a post older than the date), `known_id` (this page reached the post you already hold), `end` (the list ran out first), or null (not yet, keep walking with `next_cursor`). ## Other responses - `400`: Invalid request - missing or malformed parameters - `401`: Unauthorized - missing or invalid API key - `402`: Payment required - the account balance is too low, or the calling key has spent its own per-key credit limit - `404`: Not found - the endpoint does not exist, or the requested resource was not found upstream - `405`: Method not allowed - wrong HTTP verb for this endpoint - `409`: Conflict - an idempotent request is still in flight, its completed body exceeded the replay limit, or the cohort resource is not in a state that allows this operation - `410`: Gone - the discover run's retention passed and its rows were deleted; start a new run - `413`: Payload too large: the JSON request body exceeds the 1 MB size limit and is rejected before parsing, or a single cohort result cannot fit beneath the 1 MB response-page ceiling - `422`: Idempotency payload mismatch - this Idempotency-Key was already used with a different request payload; or a discover request with async=false is over what one call can answer (NEEDS_ASYNC) - `429`: Rate or concurrency limit exceeded. `RATE_LIMITED`: more than 600 requests in a 1-minute sliding window on this API key (headers `X-RateLimit-Limit`/`Remaining`/`Reset`; `Retry-After` is seconds until the window resets). `CONCURRENCY_LIMIT`: more than 50 simultaneous in-flight requests (headers `X-Concurrency-Limit`/`Remaining`; short static `Retry-After`). Both are unbilled. Honor `Retry-After`, then back off with jitter. See /docs/rate-limits. - `500`: Internal server error - an immediate refund is attempted; credits_used reports the committed net charge and exact compensation is queued if that refund cannot commit - `502`: Upstream error - the platform returned an error; an immediate refund is attempted and credits_used reports the committed net charge - `503`: Service unavailable - the request is unbilled when rejected before deduction, otherwise an immediate refund is attempted and credits_used reports the committed net charge - `504`: Gateway timeout - the request was still running close to the platform time limit and is answered as UPSTREAM_ERROR, the same code as 502. An immediate refund is attempted and credits_used reports the committed net charge. When details.retry_with is set, retryable is false and there is no Retry-After --- [API](https://skmtc.dev/socialcrawl/apis/socialcrawl-api.md) · [All operations](https://skmtc.dev/socialcrawl/apis/socialcrawl-api/llms.txt) · [OpenAPI document](https://skmtc.dev/socialcrawl/apis/socialcrawl-api/revisions/a0ad2511004a?raw)