Bulk Sync

Create Bulk Sync

Changed on

Creates a new bulk sync.

Bulk syncs are used for the ELT pattern (Extract, Load, and Transform), where you want to sync un-transformed data to your data warehouses, databases, or cloud storage buckets like S3.

All of the functionality described in the product documentation is configurable via the API.

Sample code examples:

  • Bulk sync (ELT) from Salesforce to S3
  • Bulk sync (ELT) from Salesforce to Snowflake
  • Bulk sync (ELT) from HubSpot to PostgreSQL

Connection specific configuration

The destination_configuration is integration-specific configuration for the selected bulk sync destination. This includes settings such as the output schema and is required when creating a new sync.

The source_configuration is optional. It allows configuration for how Polytomic reads data from the source connection. This will not be available for integrations that do not support additional configuration.

Consult the connection configurations to see configurations for particular integrations (for example, here is the available source configuration for the PostgreSQL bulk sync source).

Defaults and selection behavior

If schemas is omitted, the sync is created with all available source schemas selected. Pass schemas explicitly if you want the initial sync to include only a subset of tables or objects.

Schedule times are interpreted in UTC.

When omitted, automatic discovery defaults are conservative:

  • automatically_add_new_objects defaults to not enabling newly discovered source objects automatically.
  • automatically_add_new_fields defaults to enabling newly discovered fields on already selected objects.
  • normalize_names defaults to enabled.
post/api/bulk/syncs

Request

  • Base URL
    https://app.polytomic.com
  • Auth
    • HTTP bearer
    • or HTTP basic

Headers

X-Polytomic-Versionstring

Request body

activeboolean nullable

Whether the sync is active. Inactive syncs do not run on their schedule but can still be triggered manually.

automatically_add_new_fields'all' | 'onlyIncremental' | 'onlyNonIncremental' | 'none' nullable
automatically_add_new_objects'all' | 'onlyIncremental' | 'onlyNonIncremental' | 'none' nullable
concurrency_limitinteger nullable

Override the default concurrency limit for this sync.

data_cutoff_timestampstring date-time nullable

Global cutoff applied across schemas. Source records older than this timestamp are excluded from sync runs.

destination_configurationobject required

Destination-specific bulk sync configuration (e.g. output schema name, file format). The accepted keys depend on the destination connection type.

destination_connection_idstring uuid required

Unique identifier of the connection rows are written to

disable_record_timestampsboolean nullable

When true, Polytomic will not add its own timestamp columns to destination rows.

mode'snapshot' | 'replicate' | 'append' nullable required
namestring required

Human-readable name for the bulk sync

normalize_names'enabled' | 'disabled' | 'legacy' nullable
organization_idstring uuid nullable

Ignored for user and organization-scoped API keys, which always act in their own organization

policiesstring[] nullable

Permissions policy IDs applied to the bulk sync. On update, a non-empty list replaces the bulk sync's policies and an empty or omitted list leaves them unchanged.

resync_concurrency_limitinteger nullable

Override the default resync concurrency limit for this sync.

source_configurationobject nullable

Source-specific bulk sync configuration (e.g. replication slot name, sync lookback). The accepted keys depend on the source connection type.

source_connection_idstring uuid required

Unique identifier of the connection rows are read from

Example request

{
  "active": true,
  "additional_schedules": [
    {
      "day_of_week": "1",
      "frequency": "weekly",
      "hour": "12",
      "minute": "30",
      "selective_mode": "nonincrementalFields"
    },
    {
      "frequency": "continuous",
      "selective_mode": "incrementalFields"
    }
  ],
  "concurrency_limit": 50,
  "data_cutoff_timestamp": "2023-04-25T12:00:00Z",
  "default_schedule": {
    "day_of_month": "1",
    "day_of_week": "1",
    "hour": "12",
    "id": "248df4b7-aa70-47b8-a036-33ac447e668d",
    "minute": "30",
    "month": "1"
  },
  "destination_configuration": {
    "schema": "my_schema"
  },
  "destination_connection_id": "248df4b7-aa70-47b8-a036-33ac447e668d",
  "name": "My Bulk Sync",
  "organization_id": "248df4b7-aa70-47b8-a036-33ac447e668d",
  "policies": [
    "248df4b7-aa70-47b8-a036-33ac447e668d"
  ],
  "resync_concurrency_limit": 10,
  "schemas": [
    {
      "data_cutoff_timestamp": "2023-04-25T12:00:00Z",
      "disable_data_cutoff": true,
      "enabled": true,
      "fields": [
        {
          "enabled": true,
          "obfuscate": true,
          "user_output_name": "my_email"
        }
      ],
      "filters": [
        {
          "field_id": "field1",
          "value": "value"
        }
      ],
      "user_output_name": "my_contact"
    }
  ],
  "source_configuration": {
    "replication_slot": "slot"
  },
  "source_connection_id": "248df4b7-aa70-47b8-a036-33ac447e668d"
}

Response

OK

Example response

{
  "data": {
    "active": true,
    "additional_schedules": [
      {
        "created_at": "2023-04-25T12:00:00Z",
        "created_by": {
          "id": "12345678-1234-1234-1234-123456789012",
          "name": "John Doe",
          "type": "user"
        },
        "day_of_week": "1",
        "frequency": "weekly",
        "hour": "12",
        "minute": "30",
        "selective_mode": "nonincrementalFields",
        "updated_at": "2023-04-25T12:00:00Z",
        "updated_by": {
          "id": "12345678-1234-1234-1234-123456789012",
          "name": "John Doe",
          "type": "user"
        }
      },
      {
        "created_at": "2023-04-25T12:00:00Z",
        "created_by": {
          "id": "12345678-1234-1234-1234-123456789012",
          "name": "John Doe",
          "type": "user"
        },
        "frequency": "continuous",
        "selective_mode": "incrementalFields",
        "updated_at": "2023-04-25T12:00:00Z",
        "updated_by": {
          "id": "12345678-1234-1234-1234-123456789012",
          "name": "John Doe",
          "type": "user"
        }
      }
    ],
    "concurrency_limit": 50,
    "created_by": {
      "id": "12345678-1234-1234-1234-123456789012",
      "name": "John Doe",
      "type": "user"
    },
    "data_cutoff_timestamp": "2023-04-25T12:00:00Z",
    "default_schedule": {
      "created_by": {
        "id": "12345678-1234-1234-1234-123456789012",
        "name": "John Doe",
        "type": "user"
      },
      "day_of_month": "1",
      "day_of_week": "1",
      "hour": "12",
      "id": "248df4b7-aa70-47b8-a036-33ac447e668d",
      "minute": "30",
      "month": "1",
      "updated_by": {
        "id": "12345678-1234-1234-1234-123456789012",
        "name": "John Doe",
        "type": "user"
      }
    },
    "destination_configuration": {
      "schema": "my_schema"
    },
    "destination_connection_id": "248df4b7-aa70-47b8-a036-33ac447e668d",
    "id": "248df4b7-aa70-47b8-a036-33ac447e668d",
    "name": "My Bulk Sync",
    "organization_id": "248df4b7-aa70-47b8-a036-33ac447e668d",
    "policies": [
      "248df4b7-aa70-47b8-a036-33ac447e668d"
    ],
    "resync_concurrency_limit": 10,
    "source_configuration": {
      "replication_slot": "slot"
    },
    "source_connection_id": "248df4b7-aa70-47b8-a036-33ac447e668d",
    "updated_by": {
      "id": "12345678-1234-1234-1234-123456789012",
      "name": "John Doe",
      "type": "user"
    }
  }
}

Changes

    • breaking

      added to the request property allOf list

    • breaking

      added to the request property allOf list

    • breaking

      added to the request property allOf list

    • breaking

      added to the // request property allOf list

    • breaking

      added to the // request property allOf list

    • breaking

      added to the request property allOf list

    • breaking

      added to the request property allOf list

    • breaking

      added to the ///// request property allOf list

    • breaking

      added to the // request property allOf list

    • breaking

      the request property became not nullable

    • breaking

      the request property became not nullable

    • breaking

      the request property became not nullable

    • breaking

      the request property became not nullable

    • breaking

      removed the enum value ArrayContains of the request property /////

    • breaking

      removed the enum value ArrayDoesNotContain of the request property /////

    • breaking

      removed the enum value Between of the request property /////

    • breaking

      removed the enum value Equality of the request property /////

    • breaking

      removed the enum value False of the request property /////

    • breaking

      removed the enum value GreaterThan of the request property /////

    • breaking

      removed the enum value GreaterThanEqual of the request property /////

    • breaking

      removed the enum value InTheLast of the request property /////

    • breaking

      removed the enum value Inequality of the request property /////

    • breaking

      removed the enum value IsNotNull of the request property /////

    • breaking

      removed the enum value IsNull of the request property /////

    • breaking

      removed the enum value LessThan of the request property /////

    • breaking

      removed the enum value LessThanEqual of the request property /////

    • breaking

      removed the enum value OnOrAfter of the request property /////

    • breaking

      removed the enum value OnOrBefore of the request property /////

    • breaking

      removed the enum value RelativeOnOrAfter of the request property /////

    • breaking

      removed the enum value RelativeOnOrBefore of the request property /////

    • breaking

      removed the enum value StringContains of the request property /////

    • breaking

      removed the enum value StringDoesNotContain of the request property /////

    • breaking

      removed the enum value StringDoesNotEndWith of the request property /////

    • breaking

      removed the enum value StringDoesNotStartWith of the request property /////

    • breaking

      removed the enum value StringEndsWith of the request property /////

    • breaking

      removed the enum value StringLike of the request property /////

    • breaking

      removed the enum value StringMatchesTrimmed of the request property /////

    • breaking

      removed the enum value StringNotLike of the request property /////

    • breaking

      removed the enum value StringNotOneOf of the request property /////

    • breaking

      removed the enum value StringOneOf of the request property /////

    • breaking

      removed the enum value StringStartsWith of the request property /////

    • breaking

      removed the enum value True of the request property /////

    • breaking

      removed the enum value all of the request property

    • breaking

      removed the enum value all of the request property

    • breaking

      removed the enum value append of the request property

    • breaking

      removed the enum value builder of the request property //

    • breaking

      removed the enum value continuous of the request property //

    • breaking

      removed the enum value custom of the request property //

    • breaking

      removed the enum value daily of the request property //

    • breaking

      removed the enum value dbtcloud of the request property //

    • breaking

      removed the enum value disabled of the request property

    • breaking

      removed the enum value enabled of the request property

    • breaking

      removed the enum value hourly of the request property //

    • breaking

      removed the enum value incrementalFields of the request property //

    • breaking

      removed the enum value legacy of the request property

    • breaking

      removed the enum value manual of the request property //

    • breaking

      removed the enum value multi of the request property //

    • breaking

      removed the enum value none of the request property //

    • breaking

      removed the enum value none of the request property

    • breaking

      removed the enum value none of the request property

    • breaking

      removed the enum value nonincrementalFields of the request property //

    • breaking

      removed the enum value normal of the request property //

    • breaking

      removed the enum value onlyIncremental of the request property

    • breaking

      removed the enum value onlyIncremental of the request property

    • breaking

      removed the enum value onlyNonIncremental of the request property

    • breaking

      removed the enum value onlyNonIncremental of the request property

    • breaking

      removed the enum value rebuild of the request property //

    • breaking

      removed the enum value refetch of the request property //

    • breaking

      removed the enum value replicate of the request property

    • breaking

      removed the enum value resync of the request property //

    • breaking

      removed the enum value runafter of the request property //

    • breaking

      removed the enum value snapshot of the request property

    • breaking

      removed the enum value weekly of the request property //

    • breaking

      the // request property type changed from string to no type

    • breaking

      the // request property type changed from string to no type

    • breaking

      the // request property type changed from string to no type

    • breaking

      the request property type changed from string to no type

    • breaking

      the request property type changed from string to no type

    • breaking

      the request property type changed from object to no type

    • breaking

      the request property type changed from string to no type

    • breaking

      the request property type changed from string to no type

    • breaking

      the ///// request property type changed from string to no type

    • breaking

      the response's property type changed from object to no type for status

    • warning

      removed the request property /

    • warning

      removed the request property /

    • warning

      removed the request property /

    • warning

      removed the request property /

    • warning

      removed the request property /

    • warning

      removed the request property /

    • warning

      removed the request property /

    • warning

      removed the optional property / from the response with the status

    • warning

      removed the optional property / from the response with the status

    • warning

      removed the optional property / from the response with the status

    • warning

      removed the optional property / from the response with the status

    • warning

      removed the optional property / from the response with the status

    • warning

      removed the optional property / from the response with the status

    • warning

      removed the optional property / from the response with the status

    • warning

      removed the optional property / from the response with the status

    • warning

      removed the optional property / from the response with the status

    • warning

      removed the optional property / from the response with the status

    • warning

      removed the optional property / from the response with the status

    • warning

      removed the optional property / from the response with the status

    • warning

      removed the optional property / from the response with the status

    • warning

      removed the optional property / from the response with the status

    • warning

      removed the optional property / from the response with the status

    • warning

      removed the optional property / from the response with the status

    • warning

      removed the optional property / from the response with the status

    • warning

      removed the optional property / from the response with the status

    • warning

      removed the optional property / from the response with the status

    • warning

      removed the optional property / from the response with the status

    • warning

      removed the optional property / from the response with the status

    • warning

      removed the optional property / from the response with the status

    • warning

      removed the optional property / from the response with the status

    • info

      added to the response property allOf list for the response status

    • warning

      added the new append enum value to the / response property for the response status

    • info

      added the new append enum value to the request property