Create Bulk Sync
Changed onCreates a new bulk sync.
Bulk syncs are used for the ELT pattern (Extract, Load, and Transform), where you want to sync un-transformed data to your data warehouses, databases, or cloud storage buckets like S3.
All of the functionality described in the product documentation is configurable via the API.
Sample code examples:
- Bulk sync (ELT) from Salesforce to S3
- Bulk sync (ELT) from Salesforce to Snowflake
- Bulk sync (ELT) from HubSpot to PostgreSQL
Connection specific configuration
The destination_configuration is integration-specific configuration for the selected bulk sync destination. This includes settings such as the output schema and is required when creating a new sync.
The source_configuration is optional. It allows configuration for how Polytomic reads data from the source connection. This will not be available for integrations that do not support additional configuration.
Consult the connection configurations to see configurations for particular integrations (for example, here is the available source configuration for the PostgreSQL bulk sync source).
Defaults and selection behavior
If schemas is omitted, the sync is created with all available source schemas selected. Pass schemas explicitly if you want the initial sync to include only a subset of tables or objects.
Schedule times are interpreted in UTC.
When omitted, automatic discovery defaults are conservative:
- automatically_add_new_objects defaults to not enabling newly discovered source objects automatically.
- automatically_add_new_fields defaults to enabling newly discovered fields on already selected objects.
- normalize_names defaults to enabled.
Request
- Base URLhttps://app.polytomic.com
- Auth
- HTTP bearer
- or HTTP basic
Headers
Request body
Example request
{
"active": true,
"additional_schedules": [
{
"day_of_week": "1",
"frequency": "weekly",
"hour": "12",
"minute": "30",
"selective_mode": "nonincrementalFields"
},
{
"frequency": "continuous",
"selective_mode": "incrementalFields"
}
],
"concurrency_limit": 50,
"data_cutoff_timestamp": "2023-04-25T12:00:00Z",
"default_schedule": {
"day_of_month": "1",
"day_of_week": "1",
"hour": "12",
"id": "248df4b7-aa70-47b8-a036-33ac447e668d",
"minute": "30",
"month": "1"
},
"destination_configuration": {
"schema": "my_schema"
},
"destination_connection_id": "248df4b7-aa70-47b8-a036-33ac447e668d",
"name": "My Bulk Sync",
"organization_id": "248df4b7-aa70-47b8-a036-33ac447e668d",
"policies": [
"248df4b7-aa70-47b8-a036-33ac447e668d"
],
"resync_concurrency_limit": 10,
"schemas": [
{
"data_cutoff_timestamp": "2023-04-25T12:00:00Z",
"disable_data_cutoff": true,
"enabled": true,
"fields": [
{
"enabled": true,
"obfuscate": true,
"user_output_name": "my_email"
}
],
"filters": [
{
"field_id": "field1",
"value": "value"
}
],
"user_output_name": "my_contact"
}
],
"source_configuration": {
"replication_slot": "slot"
},
"source_connection_id": "248df4b7-aa70-47b8-a036-33ac447e668d"
}Response
OK
Example response
{
"data": {
"active": true,
"additional_schedules": [
{
"created_at": "2023-04-25T12:00:00Z",
"created_by": {
"id": "12345678-1234-1234-1234-123456789012",
"name": "John Doe",
"type": "user"
},
"day_of_week": "1",
"frequency": "weekly",
"hour": "12",
"minute": "30",
"selective_mode": "nonincrementalFields",
"updated_at": "2023-04-25T12:00:00Z",
"updated_by": {
"id": "12345678-1234-1234-1234-123456789012",
"name": "John Doe",
"type": "user"
}
},
{
"created_at": "2023-04-25T12:00:00Z",
"created_by": {
"id": "12345678-1234-1234-1234-123456789012",
"name": "John Doe",
"type": "user"
},
"frequency": "continuous",
"selective_mode": "incrementalFields",
"updated_at": "2023-04-25T12:00:00Z",
"updated_by": {
"id": "12345678-1234-1234-1234-123456789012",
"name": "John Doe",
"type": "user"
}
}
],
"concurrency_limit": 50,
"created_by": {
"id": "12345678-1234-1234-1234-123456789012",
"name": "John Doe",
"type": "user"
},
"data_cutoff_timestamp": "2023-04-25T12:00:00Z",
"default_schedule": {
"created_by": {
"id": "12345678-1234-1234-1234-123456789012",
"name": "John Doe",
"type": "user"
},
"day_of_month": "1",
"day_of_week": "1",
"hour": "12",
"id": "248df4b7-aa70-47b8-a036-33ac447e668d",
"minute": "30",
"month": "1",
"updated_by": {
"id": "12345678-1234-1234-1234-123456789012",
"name": "John Doe",
"type": "user"
}
},
"destination_configuration": {
"schema": "my_schema"
},
"destination_connection_id": "248df4b7-aa70-47b8-a036-33ac447e668d",
"id": "248df4b7-aa70-47b8-a036-33ac447e668d",
"name": "My Bulk Sync",
"organization_id": "248df4b7-aa70-47b8-a036-33ac447e668d",
"policies": [
"248df4b7-aa70-47b8-a036-33ac447e668d"
],
"resync_concurrency_limit": 10,
"source_configuration": {
"replication_slot": "slot"
},
"source_connection_id": "248df4b7-aa70-47b8-a036-33ac447e668d",
"updated_by": {
"id": "12345678-1234-1234-1234-123456789012",
"name": "John Doe",
"type": "user"
}
}
}Changes
- breaking
added to the
request propertyallOflist - breaking
added to the
request propertyallOflist - breaking
added to the
request propertyallOflist - breaking
added to the
//request propertyallOflist - breaking
added to the
//request propertyallOflist - breaking
added to the
request propertyallOflist - breaking
added to the
request propertyallOflist - breaking
added to the
/////request propertyallOflist - breaking
added to the
//request propertyallOflist - breaking
the request property
became not nullable - breaking
the request property
became not nullable - breaking
the request property
became not nullable - breaking
the request property
became not nullable - breaking
removed the enum value
ArrayContainsof the request property///// - breaking
removed the enum value
ArrayDoesNotContainof the request property///// - breaking
removed the enum value
Betweenof the request property///// - breaking
removed the enum value
Equalityof the request property///// - breaking
removed the enum value
Falseof the request property///// - breaking
removed the enum value
GreaterThanof the request property///// - breaking
removed the enum value
GreaterThanEqualof the request property///// - breaking
removed the enum value
InTheLastof the request property///// - breaking
removed the enum value
Inequalityof the request property///// - breaking
removed the enum value
IsNotNullof the request property///// - breaking
removed the enum value
IsNullof the request property///// - breaking
removed the enum value
LessThanof the request property///// - breaking
removed the enum value
LessThanEqualof the request property///// - breaking
removed the enum value
OnOrAfterof the request property///// - breaking
removed the enum value
OnOrBeforeof the request property///// - breaking
removed the enum value
RelativeOnOrAfterof the request property///// - breaking
removed the enum value
RelativeOnOrBeforeof the request property///// - breaking
removed the enum value
StringContainsof the request property///// - breaking
removed the enum value
StringDoesNotContainof the request property///// - breaking
removed the enum value
StringDoesNotEndWithof the request property///// - breaking
removed the enum value
StringDoesNotStartWithof the request property///// - breaking
removed the enum value
StringEndsWithof the request property///// - breaking
removed the enum value
StringLikeof the request property///// - breaking
removed the enum value
StringMatchesTrimmedof the request property///// - breaking
removed the enum value
StringNotLikeof the request property///// - breaking
removed the enum value
StringNotOneOfof the request property///// - breaking
removed the enum value
StringOneOfof the request property///// - breaking
removed the enum value
StringStartsWithof the request property///// - breaking
removed the enum value
Trueof the request property///// - breaking
removed the enum value
allof the request property - breaking
removed the enum value
allof the request property - breaking
removed the enum value
appendof the request property - breaking
removed the enum value
builderof the request property// - breaking
removed the enum value
continuousof the request property// - breaking
removed the enum value
customof the request property// - breaking
removed the enum value
dailyof the request property// - breaking
removed the enum value
dbtcloudof the request property// - breaking
removed the enum value
disabledof the request property - breaking
removed the enum value
enabledof the request property - breaking
removed the enum value
hourlyof the request property// - breaking
removed the enum value
incrementalFieldsof the request property// - breaking
removed the enum value
legacyof the request property - breaking
removed the enum value
manualof the request property// - breaking
removed the enum value
multiof the request property// - breaking
removed the enum value
noneof the request property// - breaking
removed the enum value
noneof the request property - breaking
removed the enum value
noneof the request property - breaking
removed the enum value
nonincrementalFieldsof the request property// - breaking
removed the enum value
normalof the request property// - breaking
removed the enum value
onlyIncrementalof the request property - breaking
removed the enum value
onlyIncrementalof the request property - breaking
removed the enum value
onlyNonIncrementalof the request property - breaking
removed the enum value
onlyNonIncrementalof the request property - breaking
removed the enum value
rebuildof the request property// - breaking
removed the enum value
refetchof the request property// - breaking
removed the enum value
replicateof the request property - breaking
removed the enum value
resyncof the request property// - breaking
removed the enum value
runafterof the request property// - breaking
removed the enum value
snapshotof the request property - breaking
removed the enum value
weeklyof the request property// - breaking
the
//request property type changed fromstringto no type - breaking
the
//request property type changed fromstringto no type - breaking
the
//request property type changed fromstringto no type - breaking
the
request property type changed fromstringto no type - breaking
the
request property type changed fromstringto no type - breaking
the
request property type changed fromobjectto no type - breaking
the
request property type changed fromstringto no type - breaking
the
request property type changed fromstringto no type - breaking
the
/////request property type changed fromstringto no type - breaking
the
response's property type changed fromobjectto no type for status - warning
removed the request property
/ - warning
removed the request property
/ - warning
removed the request property
/ - warning
removed the request property
/ - warning
removed the request property
/ - warning
removed the request property
/ - warning
removed the request property
/ - warning
removed the optional property
/from the response with the status - warning
removed the optional property
/from the response with the status - warning
removed the optional property
/from the response with the status - warning
removed the optional property
/from the response with the status - warning
removed the optional property
/from the response with the status - warning
removed the optional property
/from the response with the status - warning
removed the optional property
/from the response with the status - warning
removed the optional property
/from the response with the status - warning
removed the optional property
/from the response with the status - warning
removed the optional property
/from the response with the status - warning
removed the optional property
/from the response with the status - warning
removed the optional property
/from the response with the status - warning
removed the optional property
/from the response with the status - warning
removed the optional property
/from the response with the status - warning
removed the optional property
/from the response with the status - warning
removed the optional property
/from the response with the status - warning
removed the optional property
/from the response with the status - warning
removed the optional property
/from the response with the status - warning
removed the optional property
/from the response with the status - warning
removed the optional property
/from the response with the status - warning
removed the optional property
/from the response with the status - warning
removed the optional property
/from the response with the status - warning
removed the optional property
/from the response with the status - info
added to the
response propertyallOflist for the response status
- breaking
- warning
added the new
appendenum value to the/response property for the response status - info
added the new
appendenum value to the request property
- warning