Sources

Connects a new data source to an agent's knowledge base. You can add content via sitemap URL, file upload, or integration connection (Google Drive, SharePoint, etc.). The system will begin indexing the content immediately.

Notes: For large sources, indexing may take several minutes. Check the source status to monitor progress.

post/api/v1/projects/{projectId}/sources

Request

Path parameters

projectIdinteger required

The unique identifier of the agent (formerly known as project).

Request body multipart/form-data

sitemap_pathstring

Supports YouTube, Vimeo, sitemaps, and websites.

type'sitemap' | 'web-images' | 'upload'

The source type. Use web-images with image extraction modes such as images-sitemap or images-opengraph.

file_data_retensionboolean

File data retension

is_ocr_enabled0 | 1 | 2

OCR processing mode: 0 = Disabled, 1 = OCR enabled, 2 = AI Vision

is_anonymizedboolean

Anonymized

filestring binary

Single file upload as a data source. Discouraged — use files[] instead for new integrations.

Supported types include documents, audio files, and images. Images require is_vision_enabled: true on the agent. Max file size: 100 MB.

files[]string[]

Multiple file uploads. Submit each file as a separate files[] part in multipart/form-data. Up to 50 files per request, max 100MB each, max 1GB total batch size.

is_vision_enabledboolean

Whether to enable vision processing for images in the file source.

is_vision_compress_imageboolean

Whether to compress images for vision processing.

image_extraction_type'none' | 'sync_from_sitemap' | 'extract_opengraph_images'

The method used for extracting images from the website.

sync_all_imagesboolean

When extracting images from a sitemap, sync every image on a page instead of only the first one. Each synced image consumes one vision credit.

is_low_performance_mode_enabledboolean

Use a lower-performance crawl mode that is less likely to be blocked by websites with aggressive bot protection.

allow_extra_timeoutboolean

Allow extra processing time when crawling slow or large sources.

use_full_url_crawlboolean

Crawl every URL discovered on the site rather than relying solely on the sitemap.

Example request

{
  "sitemap_path": "https://example.com/sitemap.xml",
  "type": "sitemap",
  "file_data_retension": true,
  "file": "file.pdf",
  "is_vision_enabled": true,
  "is_vision_compress_image": true,
  "image_extraction_type": "sync_from_sitemap"
}

Response

Successfully created a new agent source.

status'error' | 'success'

The status of the response

Example response

{
  "status": "success",
  "data": {
    "id": 1,
    "created_at": "2021-01-01 00:00:00",
    "updated_at": "2021-01-01 00:00:00",
    "type": "sitemap",
    "sub_type": "images-opengraph",
    "settings": {
      "executive_js": true,
      "data_refresh_frequency": "never",
      "create_new_pages": true,
      "refresh_existing_pages": "never",
      "sitemap_path": "https://example.com/sitemap.xml",
      "image_extraction_type": "sync_from_sitemap"
    },
    "pages": [
      {
        "id": 1,
        "page_url": "https://example.com",
        "page_url_hash": "d41d8cd98f00b204e9800998ecf8427e",
        "project_id": 1,
        "s3_path": "project-1/page-1/file.pdf",
        "crawl_status": "queued",
        "index_status": "queued",
        "is_file": true,
        "is_refreshable": true,
        "is_file_kept": true,
        "filename": "file.pdf",
        "filesize": 100,
        "created_at": "2021-01-01 00:00:00",
        "updated_at": "2021-01-01 00:00:00",
        "deleted_at": "2021-01-01 00:00:00"
      }
    ]
  }
}

Changes

    • breaking

      request property was restricted to a list of enum values

    • breaking

      the request property type changed from boolean to integer

    • warning

      added the new 422.00 enum value to the / response property for the response status

    • warning

      added the new 422.00 enum value to the / response property for the response status

    • warning

      added the new 422.00 enum value to the / response property for the response status

    • warning

      added the new 422.00 enum value to the / response property for the response status

    • warning

      added the new advanced enum value to the // response property for the response status

    • warning

      added the new always enum value to the // response property for the response status

    • warning

      added the new daily enum value to the // response property for the response status

    • warning

      added the new if_updated enum value to the // response property for the response status

    • warning

      added the new monthly enum value to the // response property for the response status

    • warning

      added the new never enum value to the // response property for the response status

    • warning

      added the new never enum value to the // response property for the response status

    • warning

      added the new web-images enum value to the / response property for the response status

    • warning

      added the new weekly enum value to the // response property for the response status

    • info

      added the new optional request property

    • info

      added the new optional request property

    • info

      added the new optional request property

    • info

      added the new optional request property

    • info

      added the new optional request property

    • info

      added the new optional request property

    • info

      added the new optional request property

    • info

      added the new optional request property

    • info

      added the new optional request property

    • info

      added the new 0.00 enum value to the request property

    • info

      added the new 1.00 enum value to the request property

    • info

      added the new 2.00 enum value to the request property

    • info

      added the non-success response with the status

    • info

      added the optional property // to the response with the status

    • info

      added the optional property // to the response with the status

    • info

      added the optional property // to the response with the status

    • info

      added the optional property / to the response with the status

    • info

      removed the never always if_updated enum value from the // response property for the response status

    • info

      removed the never daily weekly monthly advanced enum value from the // response property for the response status

    • breaking

      the /// response's property format changed from binary to no format for status

    • warning

      removed the optional property // from the response with the status

    • warning

      added the new Agent id is required enum value to the / response property for the response status

    • warning

      added the new Agent with id {projectId} not found enum value to the / response property for the response status

    • warning

      added the new never daily weekly monthly advanced enum value to the // response property for the response status

    • info

      added the new optional request property

    • info

      added the new optional request property

    • info

      added the optional property /// to the response with the status

    • info

      added the optional property // to the response with the status

    • info

      added the optional property // to the response with the status

    • info

      added the optional property // to the response with the status

    • info

      removed the Our system could not detect any pages in your sitemap enum value from the / response property for the response status

    • info

      removed the Our system could not validate your sitemap enum value from the / response property for the response status

    • info

      removed the Project id is required enum value from the / response property for the response status

    • info

      removed the Project with id {projectId} not found enum value from the / response property for the response status

    • info

      removed the Sitemap URL is empty enum value from the / response property for the response status