COcontext
Batch

Submit a batch

Changed on

Scrape 25K URLs or crawl large websites asynchronously.

post/batch/submit

Request

  • Base URL
    https://api.context.dev/v1
  • URL
    https://api.context.dev/v1/batch/submit
  • Auth
    HTTP bearer

Headers

Idempotency-Keystring

Any string unique to this submission. Retries with the same key return the original batch.

Request body

webhookUrlstring

URL notified when the batch finishes.

tagsstring[]

Tags stored on the batch. Filter the batch list by them later.

Example request

{
  "input": {
    "data": {
      "urls": [
        {
          "url": "https://example.com/products/anvil",
          "itemId": "sku-1"
        }
      ],
      "options": {
        "country": "de"
      }
    }
  },
  "tags": [
    "docs",
    "competitor"
  ]
}

Response

Batch accepted. Read progress and results from GET /batch/{batch_id}.

idstring required

Batch ID. Poll GET /batch/{batch_id} with it.

status'queued' required

Always queued. An accepted batch has not started yet.

mode'scrape' | 'crawl' required

How pages will be selected.

format'markdown' | 'html' required

What each page will be returned as.

tagsstring[] required

Tags stored on the batch.

created_atstring required

When the batch was created.

webhook_secretstring

Signing secret for the completion webhook, returned only here and never again. Store it now; it is not repeated by GET /batch/{batch_id}.

Example response

{
  "id": "batch_9f2c8a",
  "tags": [
    "docs"
  ],
  "crawl": {
    "source": {
      "url": "https://example.com/docs"
    },
    "max_pages": 500,
    "url_pattern": "^https://example\\.com/docs/"
  },
  "input": {
    "reserved": 24817,
    "submitted": 25000,
    "duplicates": 183
  },
  "credits": {
    "reserved": 24817
  },
  "invalid_urls": [
    {
      "reason": "Must be a public http:// or https:// URL"
    }
  ]
}

Changes

  • 1febe5476266Revision changes
    • info

      added the required property / to the response with the status

  • 9eddb9e1a8eaRevision changes
    • breaking

      request property ////// list-of-types was narrowed by removing types string from media type application/json

    • breaking

      request property ////// list-of-types was narrowed by removing types string from media type application/json

    • breaking

      request property ////// list-of-types was narrowed by removing types string from media type application/json

    • breaking

      request property ////// list-of-types was narrowed by removing types string from media type application/json

    • breaking

      request property ////// list-of-types was narrowed by removing types string from media type application/json

    • breaking

      request property ////// list-of-types was narrowed by removing types string from media type application/json

    • breaking

      request property ////// list-of-types was narrowed by removing types string from media type application/json

    • breaking

      request property ////// list-of-types was narrowed by removing types string from media type application/json