Sources

Update source settings.

Changed on

Updates configuration for a data source, such as auto-sync frequency, crawl depth, or file filters. Use this to fine-tune how the source is indexed and kept up to date.

put/api/v1/projects/{projectId}/sources/{sourceId}

Request

Path parameters

projectIdinteger required

The unique identifier of the agent (formerly known as project).

sourceIdinteger required

The unique identifier of the source to update.

Request body

executive_jsboolean

Whether the agent source should execute JavaScript

data_refresh_frequency'never' | 'daily' | 'weekly' | 'monthly' | 'advanced'

How often the source content is re-fetched and re-indexed.

Values:

  • never: no automatic refresh.
  • daily / weekly / monthly: refresh on that cadence.
  • advanced: uses either custom schedule field.
create_new_pagesboolean

Add new pages to agent automatically during refresh agent source

remove_unexist_pagesboolean

Remove pages from agent automatically during refresh agent source

refresh_existing_pages'never' | 'always' | 'if_updated'

Refresh existing page during refresh agent source

allow_extra_timeoutboolean

Allow extra timeout for agent source

image_extraction_type'none' | 'sync_from_sitemap' | 'extract_opengraph_images'

The method used for extracting images from the website.

update_existing_imagesboolean

Whether to update existing images during refresh.

sync_all_imagesboolean

When extracting images from a sitemap, sync every image on a page instead of only the first one. Each synced image consumes one vision credit.

Example request

{
  "executive_js": true,
  "data_refresh_frequency": "never",
  "create_new_pages": true,
  "refresh_existing_pages": "never",
  "refresh_schedule": {
    "days": [
      1,
      3,
      5
    ],
    "hours": [
      "08:00",
      "23:59"
    ]
  },
  "advanced_refresh_schedule": {
    "version": 2,
    "times_by_day": {
      "1": [
        "07:00"
      ]
    }
  },
  "image_extraction_type": "sync_from_sitemap"
}

Response

Successfully updated agent source.

status'error' | 'success'

The status of the response

Example response

{
  "status": "success",
  "data": {
    "id": 1,
    "created_at": "2021-01-01 00:00:00",
    "updated_at": "2021-01-01 00:00:00",
    "type": "sitemap",
    "sub_type": "images-opengraph",
    "settings": {
      "executive_js": true,
      "data_refresh_frequency": "never",
      "create_new_pages": true,
      "refresh_existing_pages": "never",
      "sitemap_path": "https://example.com/sitemap.xml",
      "image_extraction_type": "sync_from_sitemap"
    },
    "pages": [
      {
        "id": 1,
        "page_url": "https://example.com",
        "page_url_hash": "d41d8cd98f00b204e9800998ecf8427e",
        "project_id": 1,
        "s3_path": "project-1/page-1/file.pdf",
        "crawl_status": "queued",
        "index_status": "queued",
        "is_file": true,
        "is_refreshable": true,
        "is_file_kept": true,
        "filename": "file.pdf",
        "filesize": 100,
        "created_at": "2021-01-01 00:00:00",
        "updated_at": "2021-01-01 00:00:00",
        "deleted_at": "2021-01-01 00:00:00"
      }
    ]
  }
}

Changes

    • breaking

      the request property became not nullable

    • breaking

      removed the enum value never always if_updated of the request property

    • breaking

      removed the enum value never daily weekly monthly advanced of the request property

    • breaking

      the request property type changed from array to object

    • warning

      added the new 422.00 enum value to the / response property for the response status

    • warning

      added the new 422.00 enum value to the / response property for the response status

    • warning

      added the new 422.00 enum value to the / response property for the response status

    • warning

      added the new 422.00 enum value to the / response property for the response status

    • warning

      added the new advanced enum value to the // response property for the response status

    • warning

      added the new always enum value to the // response property for the response status

    • warning

      added the new daily enum value to the // response property for the response status

    • warning

      added the new if_updated enum value to the // response property for the response status

    • warning

      added the new monthly enum value to the // response property for the response status

    • warning

      added the new never enum value to the // response property for the response status

    • warning

      added the new never enum value to the // response property for the response status

    • warning

      added the new web-images enum value to the / response property for the response status

    • warning

      added the new weekly enum value to the // response property for the response status

    • info

      added the new optional request property

    • info

      added the new optional request property

    • info

      added the new optional request property

    • info

      added the new optional request property /

    • info

      added the new optional request property /

    • info

      added the new optional request property

    • info

      added the new optional request property

    • info

      the request property default value changed from to {}

    • info

      added the new advanced enum value to the request property

    • info

      added the new always enum value to the request property

    • info

      added the new daily enum value to the request property

    • info

      added the new if_updated enum value to the request property

    • info

      added the new monthly enum value to the request property

    • info

      added the new never enum value to the request property

    • info

      added the new never enum value to the request property

    • info

      added the new weekly enum value to the request property

    • info

      added the non-success response with the status

    • info

      added the optional property // to the response with the status

    • info

      added the optional property // to the response with the status

    • info

      added the optional property // to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      removed the never always if_updated enum value from the // response property for the response status

    • info

      removed the never daily weekly monthly advanced enum value from the // response property for the response status

    • info

      endpoint added