---
title: "Update a Source"
method: PATCH
path: "/sources/{sourceId}"
tags: ["public_sources", "public", "Sources"]
---
# Update a Source
`PATCH /sources/{sourceId}`
## Request body
- SourcePatchRequest
- `name` string
- `workspaceId` string, uuid
- `configuration` union — The values required to configure the source.
- SourceAha
- `api_key` string, required — API Key
- `url` string, required — URL
- `sourceType` 'aha', required
- Source7shifts
- `access_token` string, required — Access token to use for authentication. Generate it in the 7shifts Developer Tools.
- `start_date` string, date-time, required
- `sourceType` '7shifts', required
- SourceActivecampaign
- `api_key` string, required — API Key
- `account_username` string, required — Account Username
- `sourceType` 'activecampaign', required
- SourceAirbyte
- `client_id` string, required
- `start_date` string, date-time, required
- `client_secret` string, required
- `sourceType` 'airbyte', required
- SourceAircall
- `api_id` string, required — App ID found at settings https://dashboard.aircall.io/integrations/api-keys
- `api_token` string, required — App token found at settings (Ref- https://dashboard.aircall.io/integrations/api-keys)
- `start_date` string, date-time, required — Date time filter for incremental filter, Specify which date to extract from.
- `sourceType` 'aircall', required
- SourceAirtable
- `credentials` union
- object
- `auth_method` 'oauth2.0'
- `client_id` string, required — The client ID of the Airtable developer application.
- `client_secret` string, required — The client secret the Airtable developer application.
- `access_token` string — Access Token for making authenticated requests.
- `token_expiry_date` string, date-time — The date-time when the access token should be refreshed.
- `refresh_token` string, required — The key to refresh the expired access token.
- object
- `auth_method` 'api_key'
- `api_key` string, required — The Personal Access Token for the Airtable account. See the Support Guide for more information on how to obtain this token.
- `sourceType` 'airtable'
- SourceAlgolia
- `api_key` string, required
- `application_id` string, required — The application ID for your application found in settings
- `search_query` string — Search query to be used with indexes_query stream with format defined in `https://www.algolia.com/doc/rest-api/search/#tag/Search/operation/searchSingleIndex`
- `start_date` string, date-time, required
- `object_id` string — Object ID within index for search queries
- `sourceType` 'algolia', required
- SourceAmazonAds
- `auth_type` 'oauth2.0'
- `client_id` string, required — The client ID of your Amazon Ads developer application. See the docs for more information.
- `client_secret` string, required — The client secret of your Amazon Ads developer application. See the docs for more information.
- `refresh_token` string, required — Amazon Ads refresh token. See the docs for more information on how to obtain this token.
- `region` 'NA' | 'EU' | 'FE' — Region to pull data from (EU/NA/FE). See docs for more details.
- `start_date` string, date — The Start date for collecting reports, should not be more than 60 days in the past. In YYYY-MM-DD format
- `profiles` integer[] — Profile IDs you want to fetch data for. The Amazon Ads source connector supports only profiles with seller and vendor type, profiles with agency type will be ignored. See docs for more details. Note: If Marketplace IDs are also selected, profiles will be selected if they match the Profile ID OR the Marketplace ID.
- `marketplace_ids` string[] — Marketplace IDs you want to fetch data for. Note: If Profile IDs are also selected, profiles will be selected if they match the Profile ID OR the Marketplace ID.
- `state_filter` string[] — Reflects the state of the Display, Product, and Brand Campaign streams as enabled, paused, or archived. If you do not populate this field, it will be ignored completely.
- `look_back_window` integer — The amount of days to go back in time to get the updated data from Amazon Ads
- `report_record_types` string[] — Optional configuration which accepts an array of string of record types. Leave blank for default behaviour to pull all report types. Use this config option only if you want to pull specific report type(s). See docs for more details
- `sourceType` 'amazon-ads', required
- SourceAmazonSellerPartner
- `auth_type` 'oauth2.0'
- `aws_environment` 'PRODUCTION' | 'SANDBOX', required — Select the AWS Environment.
- `region` 'AE' | 'AU' | 'BE' | 'BR' | 'CA' | 'DE' | 'EG' | 'ES' | 'FR' | 'GB' | 'IN' | 'IT' | 'JP' | 'MX' | 'NL' | 'PL' | 'SA' | 'SE' | 'SG' | 'TR' | 'UK' | 'US', required — Select the AWS Region.
- `account_type` 'Seller' | 'Vendor', required — Type of the Account you're going to authorize the Airbyte application by
- `lwa_app_id` string, required — Your Login with Amazon Client ID.
- `lwa_client_secret` string, required — Your Login with Amazon Client Secret.
- `refresh_token` string, required — The Refresh Token obtained via OAuth flow authorization.
- `replication_start_date` string, date-time — UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated. If start date is not provided or older than 2 years ago from today, the date 2 years ago from today will be used.
- `replication_end_date` string, date-time — UTC date and time in the format 2017-01-25T00:00:00Z. Any data after this date will not be replicated.
- `period_in_days` integer — For syncs spanning a large date range, this option is used to request data in a smaller fixed window to improve sync reliability. This time window can be configured granularly by day.
- `report_options_list` object[] — Additional information passed to reports. This varies by report type.
- `report_name` 'GET_AFN_INVENTORY_DATA' | 'GET_AFN_INVENTORY_DATA_BY_COUNTRY' | 'GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL' | 'GET_FBA_ESTIMATED_FBA_FEES_TXT_DATA' | 'GET_FBA_FULFILLMENT_CUSTOMER_RETURNS_DATA' | 'GET_FBA_FULFILLMENT_CUSTOMER_SHIPMENT_PROMOTION_DATA' | 'GET_FBA_FULFILLMENT_CUSTOMER_SHIPMENT_REPLACEMENT_DATA' | 'GET_FBA_FULFILLMENT_REMOVAL_ORDER_DETAIL_DATA' | 'GET_FBA_FULFILLMENT_REMOVAL_SHIPMENT_DETAIL_DATA' | 'GET_FBA_INVENTORY_PLANNING_DATA' | 'GET_FBA_MYI_UNSUPPRESSED_INVENTORY_DATA' | 'GET_FBA_REIMBURSEMENTS_DATA' | 'GET_FBA_SNS_FORECAST_DATA' | 'GET_FBA_SNS_PERFORMANCE_DATA' | 'GET_FBA_STORAGE_FEE_CHARGES_DATA' | 'GET_FLAT_FILE_ACTIONABLE_ORDER_DATA_SHIPPING' | 'GET_FLAT_FILE_ALL_ORDERS_DATA_BY_LAST_UPDATE_GENERAL' | 'GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL' | 'GET_FLAT_FILE_ARCHIVED_ORDERS_DATA_BY_ORDER_DATE' | 'GET_FLAT_FILE_OPEN_LISTINGS_DATA' | 'GET_FLAT_FILE_RETURNS_DATA_BY_RETURN_DATE' | 'GET_LEDGER_DETAIL_VIEW_DATA' | 'GET_LEDGER_SUMMARY_VIEW_DATA' | 'GET_MERCHANT_CANCELLED_LISTINGS_DATA' | 'GET_MERCHANT_LISTINGS_ALL_DATA' | 'GET_MERCHANT_LISTINGS_DATA' | 'GET_MERCHANT_LISTINGS_DATA_BACK_COMPAT' | 'GET_MERCHANT_LISTINGS_INACTIVE_DATA' | 'GET_MERCHANTS_LISTINGS_FYP_REPORT' | 'GET_ORDER_REPORT_DATA_SHIPPING' | 'GET_RESTOCK_INVENTORY_RECOMMENDATIONS_REPORT' | 'GET_SELLER_FEEDBACK_DATA' | 'GET_STRANDED_INVENTORY_UI_DATA' | 'GET_V2_SETTLEMENT_REPORT_DATA_FLAT_FILE' | 'GET_XML_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL' | 'GET_XML_BROWSE_TREE_DATA' | 'GET_VENDOR_REAL_TIME_INVENTORY_REPORT' | 'GET_BRAND_ANALYTICS_MARKET_BASKET_REPORT' | 'GET_BRAND_ANALYTICS_SEARCH_TERMS_REPORT' | 'GET_BRAND_ANALYTICS_REPEAT_PURCHASE_REPORT' | 'GET_SALES_AND_TRAFFIC_REPORT' | 'GET_VENDOR_SALES_REPORT' | 'GET_VENDOR_INVENTORY_REPORT' | 'GET_VENDOR_NET_PURE_PRODUCT_MARGIN_REPORT' | 'GET_VENDOR_TRAFFIC_REPORT', required
- `stream_name` string, required
- `options_list` object[], required — List of options
- `option_name` string, required
- `option_value` string, required
- `wait_to_avoid_fatal_errors` boolean — For report based streams with known amount of requests per time period, this option will use waiting time between requests to avoid fatal statuses in reports. See Troubleshooting section for more details
- `sourceType` 'amazon-seller-partner', required
- SourceAmazonSqs
- `queue_url` string, required — URL of the SQS Queue
- `region` 'af-south-1' | 'ap-east-1' | 'ap-northeast-1' | 'ap-northeast-2' | 'ap-northeast-3' | 'ap-south-1' | 'ap-south-2' | 'ap-southeast-1' | 'ap-southeast-2' | 'ap-southeast-3' | 'ap-southeast-4' | 'ca-central-1' | 'ca-west-1' | 'cn-north-1' | 'cn-northwest-1' | 'eu-central-1' | 'eu-central-2' | 'eu-north-1' | 'eu-south-1' | 'eu-south-2' | 'eu-west-1' | 'eu-west-2' | 'eu-west-3' | 'il-central-1' | 'me-central-1' | 'me-south-1' | 'sa-east-1' | 'us-east-1' | 'us-east-2' | 'us-gov-east-1' | 'us-gov-west-1' | 'us-west-1' | 'us-west-2', required — AWS Region of the SQS Queue
- `delete_messages` boolean, required — If Enabled, messages will be deleted from the SQS Queue after being read. If Disabled, messages are left in the queue and can be read more than once. WARNING: Enabling this option can result in data loss in cases of failure, use with caution, see documentation for more detail.
- `max_batch_size` integer — Max amount of messages to get in one batch (10 max)
- `max_wait_time` integer — Max amount of time in seconds to wait for messages in a single poll (20 max)
- `attributes_to_return` string — Comma separated list of Mesage Attribute names to return
- `visibility_timeout` integer — Modify the Visibility Timeout of the individual message from the Queue's default (seconds).
- `access_key` string — The Access Key ID of the AWS IAM Role to use for pulling messages
- `secret_key` string — The Secret Key of the AWS IAM Role to use for pulling messages
- `sourceType` 'amazon-sqs', required
- SourceAmplitude
- `data_region` 'Standard Server' | 'EU Residency Server' — Amplitude data region server
- `api_key` string, required — Amplitude API Key. See the setup guide for more information on how to obtain this key.
- `secret_key` string, required — Amplitude Secret Key. See the setup guide for more information on how to obtain this key.
- `start_date` string, date-time, required — UTC date and time in the format 2021-01-25T00:00:00Z. Any data before this date will not be replicated.
- `request_time_range` integer — According to Considerations too big time range in request can cause a timeout error. In this case, set shorter time interval in hours.
- `active_users_group_by_country` boolean — According to Considerations the grouping by `Country` is optional, if you're facing issues fetching the stream, or checking the connection please set this to `False` instead.
- `sourceType` 'amplitude', required
- SourceApifyDataset
- `token` string, required — Personal API token of your Apify account. In Apify Console, you can find your API token in the Settings section under the Integrations tab after you login. See the Apify Docs for more information.
- `dataset_id` string, required — ID of the dataset you would like to load to Airbyte. In Apify Console, you can view your datasets in the Storage section under the Datasets tab after you login. See the Apify Docs for more information.
- `sourceType` 'apify-dataset', required
- SourceAppcues
- `username` string, required
- `password` string
- `account_id` string, required — Account ID of Appcues found in account settings page (https://studio.appcues.com/settings/account)
- `start_date` string, date-time, required
- `sourceType` 'appcues', required
- SourceAppfigures
- `api_key` string, required
- `search_store` string — The store which needs to be searched in streams
- `start_date` string, date-time, required
- `group_by` 'network' | 'product' | 'country' | 'date' — Category term for grouping the search results
- `sourceType` 'appfigures', required
- SourceAppfollow
- `api_secret` string — API Key provided by Appfollow
- `sourceType` 'appfollow', required
- SourceAppleSearchAds
- `org_id` integer, required — The identifier of the organization that owns the campaign. Your Org Id is the same as your account in the Apple Search Ads UI.
- `end_date` string — Data is retrieved until that date (included)
- `client_id` string, required — A user identifier for the token request. See here
- `start_date` string, required — Start getting data from that date.
- `client_secret` string, required — A string that authenticates the user’s setup request. See here
- `sourceType` 'apple-search-ads', required
- SourceAsana
- `credentials` union — Choose how to authenticate to Github
- object
- `option_title` 'OAuth Credentials' — OAuth Credentials
- `client_id` string, required
- `client_secret` string, required
- `refresh_token` string, required
- object
- `option_title` 'PAT Credentials' — PAT Credentials
- `personal_access_token` string, required — Asana Personal Access Token (generate yours here).
- `organization_export_ids` unknown[] — Globally unique identifiers for the organization exports
- unknown
- `sourceType` 'asana'
- SourceAshby
- `api_key` string, required — The Ashby API Key, see doc here.
- `start_date` string, required — UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.
- `sourceType` 'ashby', required
- SourceAuth0
- `base_url` string, required — The Authentication API is served over HTTPS. All URLs referenced in the documentation have the following base `https://YOUR_DOMAIN`
- `credentials` union, required
- object
- `auth_type` 'oauth2_confidential_application', required
- `client_id` string, required — Your application's Client ID. You can find this value on the application's settings tab after you login the admin portal.
- `client_secret` string, required — Your application's Client Secret. You can find this value on the application's settings tab after you login the admin portal.
- `audience` string, required — The audience for the token, which is your API. You can find this in the Identifier field on your API's settings tab
- object
- `auth_type` 'oauth2_access_token', required
- `access_token` string, required — Also called API Access Token The access token used to call the Auth0 Management API Token. It's a JWT that contains specific grant permissions knowns as scopes.
- `start_date` string — UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.
- `sourceType` 'auth0', required
- SourceAwsCloudtrail
- `aws_key_id` string, required — AWS CloudTrail Access Key ID. See the docs for more information on how to obtain this key.
- `aws_secret_key` string, required — AWS CloudTrail Access Key ID. See the docs for more information on how to obtain this key.
- `aws_region_name` string, required — The default AWS Region to use, for example, us-west-1 or us-west-2. When specifying a Region inline during client initialization, this property is named region_name.
- `start_date` string, date — The date you would like to replicate data. Data in AWS CloudTrail is available for last 90 days only. Format: YYYY-MM-DD.
- `lookup_attributes_filter` object
- `attribute_key` string, required
- `attribute_value` string, required
- `sourceType` 'aws-cloudtrail', required
- SourceAzureBlobStorage — NOTE: When this Spec is changed, legacy_config_transformer.py must also be modified to uptake the changes because it is responsible for converting legacy Azure Blob Storage v0 configs into v1 configs using the File-Based CDK.
- `start_date` string, date-time — UTC date and time in the format 2017-01-25T00:00:00.000000Z. Any file modified before this date will not be replicated.
- `streams` object[], required — Each instance of this configuration defines a stream. Use this to define which files belong in the stream, their format, and how they should be parsed and validated. When sending data to warehouse destination such as Snowflake or BigQuery, each stream is a separate table.
- `name` string, required — The name of the stream.
- `globs` string[] — The pattern used to specify which files should be selected from the file system. For more information on glob pattern matching look here.
- `validation_policy` 'Emit Record' | 'Skip Record' | 'Wait for Discover' — The name of the validation policy that dictates sync behavior when a record does not adhere to the stream schema.
- `input_schema` string — The schema that will be used to validate records extracted from the file. This will override the stream schema that is auto-detected from incoming files.
- `days_to_sync_if_history_is_full` integer — When the state history of the file store is full, syncs will only read files that were last modified in the provided day range.
- `format` union, required — The configuration options that are used to alter how to read incoming files that deviate from the standard formatting.
- object
- `filetype` 'avro', required
- `double_as_string` boolean — Whether to convert double fields to strings. This is recommended if you have decimal numbers with a high degree of precision because there can be a loss precision when handling floating point numbers.
- object
- `filetype` 'csv', required
- `delimiter` string — The character delimiting individual cells in the CSV data. This may only be a 1-character string. For tab-delimited data enter '\t'.
- `quote_char` string — The character used for quoting CSV values. To disallow quoting, make this field blank.
- `escape_char` string — The character used for escaping special characters. To disallow escaping, leave this field blank.
- `encoding` string — The character encoding of the CSV data. Leave blank to default to UTF8. See list of python encodings for allowable options.
- `double_quote` boolean — Whether two quotes in a quoted CSV value denote a single quote in the data.
- `null_values` string[] — A set of case-sensitive strings that should be interpreted as null values. For example, if the value 'NA' should be interpreted as null, enter 'NA' in this field.
- `strings_can_be_null` boolean — Whether strings can be interpreted as null values. If true, strings that match the null_values set will be interpreted as null. If false, strings that match the null_values set will be interpreted as the string itself.
- `skip_rows_before_header` integer — The number of rows to skip before the header row. For example, if the header row is on the 3rd row, enter 2 in this field.
- `skip_rows_after_header` integer — The number of rows to skip after the header row.
- `header_definition` union — How headers will be defined. `User Provided` assumes the CSV does not have a header row and uses the headers provided and `Autogenerated` assumes the CSV does not have a header row and the CDK will generate headers using for `f{i}` where `i` is the index starting from 0. Else, the default behavior is to use the header from the CSV file. If a user wants to autogenerate or provide column names for a CSV having headers, they can skip rows.
- …
- `true_values` string[] — A set of case-sensitive strings that should be interpreted as true values.
- `false_values` string[] — A set of case-sensitive strings that should be interpreted as false values.
- `ignore_errors_on_fields_mismatch` boolean — Whether to ignore errors that occur when the number of fields in the CSV does not match the number of columns in the schema.
- object
- `filetype` 'jsonl', required
- object
- `filetype` 'parquet', required
- `decimal_as_float` boolean — Whether to convert decimal fields to floats. There is a loss of precision when converting decimals to floats, so this is not recommended.
- object — Extract text from document formats (.pdf, .docx, .md, .pptx) and emit as one record per file.
- `filetype` 'unstructured', required
- `skip_unprocessable_files` boolean — If true, skip files that cannot be parsed and pass the error message along as the _ab_source_file_parse_error field. If false, fail the sync.
- `strategy` 'auto' | 'fast' | 'ocr_only' | 'hi_res' — The strategy used to parse documents. `fast` extracts text directly from the document which doesn't work for all files. `ocr_only` is more reliable, but slower. `hi_res` is the most reliable, but requires an API key and a hosted instance of unstructured and can't be used with local mode. See the unstructured.io documentation for more details: https://unstructured-io.github.io/unstructured/core/partition.html#partition-pdf
- `processing` object — Process files locally, supporting `fast` and `ocr` modes. This is the default option.
- …
- `schemaless` boolean — When enabled, syncs will not validate or structure records against the stream's schema.
- `credentials` union, required — Credentials for connecting to the Azure Blob Storage
- object
- `auth_type` 'oauth2', required
- `tenant_id` string, required — Tenant ID of the Microsoft Azure Application user
- `client_id` string, required — Client ID of your Microsoft developer application
- `client_secret` string, required — Client Secret of your Microsoft developer application
- `refresh_token` string, required — Refresh Token of your Microsoft developer application
- object
- `auth_type` 'storage_account_key', required
- `azure_blob_storage_account_key` string, required — The Azure blob storage account key.
- `azure_blob_storage_account_name` string, required — The account's name of the Azure Blob Storage.
- `azure_blob_storage_container_name` string, required — The name of the Azure blob storage container.
- `azure_blob_storage_endpoint` string — This is Azure Blob Storage endpoint domain name. Leave default value (or leave it empty if run container from command line) to use Microsoft native from example.
- `sourceType` 'azure-blob-storage', required
- SourceAzureTable
- `storage_account_name` string, required — The name of your storage account.
- `storage_access_key` string, required — Azure Table Storage Access Key. See the docs for more information on how to obtain this key.
- `storage_endpoint_suffix` string — Azure Table Storage service account URL suffix. See the docs for more information on how to obtain endpoint suffix
- `sourceType` 'azure-table', required
- SourceBambooHr
- `api_key` string, required — Api key of bamboo hr
- `subdomain` string, required — Sub Domain of bamboo hr
- `custom_reports_fields` string — Comma-separated list of fields to include in custom reports.
- `custom_reports_include_default_fields` boolean — If true, the custom reports endpoint will include the default fields defined here: https://documentation.bamboohr.com/docs/list-of-field-names.
- `start_date` string, date-time
- `sourceType` 'bamboo-hr', required
- SourceBasecamp
- `account_id` number, required
- `start_date` string, date-time, required
- `client_id` string, required
- `client_secret` string, required
- `client_refresh_token_2` string, required
- `sourceType` 'basecamp', required
- SourceBeamer
- `api_key` string, required
- `start_date` string, date-time, required
- `sourceType` 'beamer', required
- SourceBigquery
- `project_id` string, required — The GCP project ID for the project containing the target BigQuery dataset.
- `dataset_id` string — The dataset ID to search for tables and views. If you are only loading data from one dataset, setting this option could result in much faster schema discovery.
- `credentials_json` string, required — The contents of your Service Account Key JSON file. See the docs for more information on how to obtain this key.
- `sourceType` 'bigquery', required
- SourceBingAds
- `auth_method` 'oauth2.0'
- `tenant_id` string — The Tenant ID of your Microsoft Advertising developer application. Set this to "common" unless you know you need a different value.
- `client_id` string, required — The Client ID of your Microsoft Advertising developer application.
- `client_secret` string — The Client Secret of your Microsoft Advertising developer application.
- `refresh_token` string, required — Refresh Token to renew the expired Access Token.
- `developer_token` string, required — Developer token associated with user. See more info in the docs.
- `account_names` object[] — Predicates that will be used to sync data by specific accounts.
- `operator` 'Contains' | 'Equals', required — An Operator that will be used to filter accounts. The Contains predicate has features for matching words, matching inflectional forms of words, searching using wildcard characters, and searching using proximity. The Equals is used to return all rows where account name is equal(=) to the string that you provided
- `name` string, required — Account Name is a string value for comparing with the specified predicate.
- `reports_start_date` string, date — The start date from which to begin replicating report data. Any data generated before this date will not be replicated in reports. This is a UTC date in YYYY-MM-DD format. If not set, data from previous and current calendar year will be replicated.
- `lookback_window` integer — Also known as attribution or conversion window. How far into the past to look for records (in days). If your conversion window has an hours/minutes granularity, round it up to the number of days exceeding. Used only for performance report streams in incremental mode without specified Reports Start Date.
- `custom_reports` object[] — You can add your Custom Bing Ads report by creating one.
- `name` string, required — The name of the custom report, this name would be used as stream name
- `reporting_object` 'AccountPerformanceReportRequest' | 'AdDynamicTextPerformanceReportRequest' | 'AdExtensionByAdReportRequest' | 'AdExtensionByKeywordReportRequest' | 'AdExtensionDetailReportRequest' | 'AdGroupPerformanceReportRequest' | 'AdPerformanceReportRequest' | 'AgeGenderAudienceReportRequest' | 'AudiencePerformanceReportRequest' | 'CallDetailReportRequest' | 'CampaignPerformanceReportRequest' | 'ConversionPerformanceReportRequest' | 'DestinationUrlPerformanceReportRequest' | 'DSAAutoTargetPerformanceReportRequest' | 'DSACategoryPerformanceReportRequest' | 'DSASearchQueryPerformanceReportRequest' | 'GeographicPerformanceReportRequest' | 'GoalsAndFunnelsReportRequest' | 'HotelDimensionPerformanceReportRequest' | 'HotelGroupPerformanceReportRequest' | 'KeywordPerformanceReportRequest' | 'NegativeKeywordConflictReportRequest' | 'ProductDimensionPerformanceReportRequest' | 'ProductMatchCountReportRequest' | 'ProductNegativeKeywordConflictReportRequest' | 'ProductPartitionPerformanceReportRequest' | 'ProductPartitionUnitPerformanceReportRequest' | 'ProductSearchQueryPerformanceReportRequest' | 'ProfessionalDemographicsAudienceReportRequest' | 'PublisherUsagePerformanceReportRequest' | 'SearchCampaignChangeHistoryReportRequest' | 'SearchQueryPerformanceReportRequest' | 'ShareOfVoiceReportRequest' | 'UserLocationPerformanceReportRequest', required — The name of the the object derives from the ReportRequest object. You can find it in Bing Ads Api docs - Reporting API - Reporting Data Objects.
- `report_columns` string[], required — A list of available report object columns. You can find it in description of reporting object that you want to add to custom report.
- `report_aggregation` string, required — A list of available aggregations.
- `sourceType` 'bing-ads', required
- SourceBitly
- `api_key` string, required
- `start_date` string, date-time, required
- `end_date` string, date-time, required
- `sourceType` 'bitly', required
- SourceBraintree
- `merchant_id` string, required — The unique identifier for your entire gateway account. See the docs for more information on how to obtain this ID.
- `public_key` string, required — Braintree Public Key. See the docs for more information on how to obtain this key.
- `private_key` string, required — Braintree Private Key. See the docs for more information on how to obtain this key.
- `start_date` string, date-time — UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.
- `environment` 'Development' | 'Sandbox' | 'Qa' | 'Production', required — Environment specifies where the data will come from.
- `sourceType` 'braintree', required
- SourceBraze
- `url` string, required — Braze REST API endpoint
- `api_key` string, required — Braze REST API key
- `start_date` string, date, required — Rows after this date will be synced
- `sourceType` 'braze', required
- SourceBreezyHr
- `api_key` string, required
- `company_id` string, required
- `sourceType` 'breezy-hr', required
- SourceBrevo
- `api_key` string, required
- `start_date` string, date-time, required
- `sourceType` 'brevo', required
- SourceBugsnag
- `auth_token` string, required — Personal auth token for accessing the Bugsnag API. Generate it in the My Account section of Bugsnag settings.
- `start_date` string, date-time, required
- `sourceType` 'bugsnag', required
- SourceBuildkite
- `api_key` string, required
- `start_date` string, date-time, required
- `sourceType` 'buildkite', required
- SourceBuzzsprout
- `api_key` string, required
- `podcast_id` string, required — Podcast ID found in `https://www.buzzsprout.com/my/profile/api`
- `start_date` string, date-time, required
- `sourceType` 'buzzsprout', required
- SourceCalendly
- `api_key` string, required — Go to Integrations → API & Webhooks to obtain your bearer token. https://calendly.com/integrations/api_webhooks
- `start_date` string, date-time, required
- `sourceType` 'calendly', required
- SourceCallrail
- `api_key` string, required — API access key
- `account_id` string, required — Account ID
- `start_date` string, required — Start getting data from that date.
- `sourceType` 'callrail', required
- SourceCampaignMonitor
- `username` string, required
- `password` string
- `start_date` string — Date from when the sync should start
- `sourceType` 'campaign-monitor', required
- SourceCanny
- `api_key` string, required — You can find your secret API key in Your Canny Subdomain > Settings > API
- `sourceType` 'canny', required
- SourceCareQualityCommission
- `api_key` string, required — Your CQC Primary Key. See https://www.cqc.org.uk/about-us/transparency/using-cqc-data#api for steps to generate one.
- `sourceType` 'care-quality-commission', required
- SourceCart
- `credentials` union
- object
- `auth_type` 'CENTRAL_API_ROUTER', required
- `user_name` string, required — Enter your application's User Name
- `user_secret` string, required — Enter your application's User Secret
- `site_id` string, required — You can determine a site provisioning site Id by hitting https://site.com/store/sitemonitor.aspx and reading the response param PSID
- object
- `auth_type` 'SINGLE_STORE_ACCESS_TOKEN', required
- `access_token` string, required — Access Token for making authenticated requests.
- `store_name` string, required — The name of Cart.com Online Store. All API URLs start with https://[mystorename.com]/api/v1/, where [mystorename.com] is the domain name of your store.
- `start_date` string, required — The date from which you'd like to replicate the data
- `sourceType` 'cart', required
- SourceChameleon
- `api_key` string, required
- `start_date` string, date-time, required
- `limit` string — Max records per page limit
- `filter` 'tour' | 'survey' | 'launcher' — Filter for using in the `segments_experiences` stream
- `end_date` string, date-time — End date for incremental sync
- `sourceType` 'chameleon', required
- SourceChargebee
- `site_api_key` string, required — Chargebee API Key. See the docs for more information on how to obtain this key.
- `site` string, required — The site prefix for your Chargebee instance.
- `start_date` string, date-time, required — UTC date and time in the format 2017-01-25T00:00:00.000Z. Any data before this date will not be replicated.
- `product_catalog` '1.0' | '2.0' — Product Catalog version of your Chargebee site. Instructions on how to find your version you may find here under `API Version` section. If left blank, the product catalog version will be set to 2.0.
- `sourceType` 'chargebee', required
- SourceChargify
- `api_key` string, required — Maxio Advanced Billing/Chargify API Key.
- `domain` string, required — Chargify domain. Normally this domain follows the following format
- `username` string, required
- `password` string
- `sourceType` 'chargify', required
- SourceChartmogul
- `api_key` string, required — Your Chartmogul API key. See the docs for info on how to obtain this.
- `start_date` string, date-time, required — UTC date and time in the format 2017-01-25T00:00:00Z. When feasible, any data before this date will not be replicated.
- `sourceType` 'chartmogul', required
- SourceCimis
- `api_key` string, required
- `targets_type` 'WSN station numbers' | 'California zip codes' | 'decimal-degree coordinates' | 'street addresses', required
- `targets` unknown[], required
- unknown
- `daily_data_items` unknown[]
- unknown
- `hourly_data_items` unknown[]
- unknown
- `start_date` string, date-time, required
- `end_date` string, date-time, required
- `unit_of_measure` 'E' | 'M'
- `sourceType` 'cimis', required
- SourceCircleci
- `api_key` string, required
- `org_id` string, required — The org ID found in `https://app.circleci.com/settings/organization/circleci/xxxxx/overview`
- `start_date` string, date-time, required
- `project_id` string, required — Project ID found in the project settings
- `job_id` string — Job ID for fetching information
- `workflow_id` string — workflow ID of a project pipeline
- `workflow_name` string — Workflow name for fetching information
- `job_number` string — Job Number of the workflow
- `sourceType` 'circleci', required
- SourceClazar
- `client_id` string, required
- `client_secret` string, required
- `sourceType` 'clazar', required
- SourceClickhouse
- `host` string, required — The host endpoint of the Clickhouse cluster.
- `port` integer, required — The port of the database.
- `database` string, required — The name of the database.
- `username` string, required — The username which is used to access the database.
- `password` string — The password associated with this username.
- `jdbc_url_params` string — Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (Eg. key1=value1&key2=value2&key3=value3). For more information read about JDBC URL parameters.
- `ssl` boolean — Encrypt data using SSL.
- `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use.
- object
- `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database
- object
- `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key
- `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel.
- `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections.
- `tunnel_user` string, required — OS-level username for logging into the jump server host.
- `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa )
- object
- `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication
- `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel.
- `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections.
- `tunnel_user` string, required — OS-level username for logging into the jump server host
- `tunnel_user_password` string, required — OS-level password for logging into the jump server host
- `sourceType` 'clickhouse', required
- SourceClickupApi
- `api_token` string, required — Every ClickUp API call required authentication. This field is your personal API token. See here.
- `include_closed_tasks` boolean — Include or exclude closed tasks. By default, they are excluded. See here.
- `sourceType` 'clickup-api', required
- SourceClockify
- `api_key` string, required — You can get your api access_key here This API is Case Sensitive.
- `api_url` string — The URL for the Clockify API. This should only need to be modified if connecting to an enterprise version of Clockify.
- `workspace_id` string, required — WorkSpace Id
- `sourceType` 'clockify', required
- SourceCloseCom
- `api_key` string, required — Close.com API key (usually starts with 'api_'; find yours here).
- `start_date` string, date — The start date to sync data; all data after this date will be replicated. Leave blank to retrieve all the data available in the account. Format: YYYY-MM-DD.
- `sourceType` 'close-com', required
- SourceCoassemble
- `user_token` string, required
- `user_id` string, required
- `sourceType` 'coassemble', required
- SourceCoda
- `auth_token` string, required — Bearer token
- `sourceType` 'coda', required
- SourceCoinApi
- `api_key` string, required — API Key
- `environment` 'sandbox' | 'production', required — The environment to use. Either sandbox or production.
- `symbol_id` string, required — The symbol ID to use. See the documentation for a list. https://docs.coinapi.io/#list-all-symbols-get
- `period` string, required — The period to use. See the documentation for a list. https://docs.coinapi.io/#list-all-periods-get
- `start_date` string, required — The start date in ISO 8601 format.
- `end_date` string — The end date in ISO 8601 format. If not supplied, data will be returned from the start date to the current time, or when the count of result elements reaches its limit.
- `limit` integer — The maximum number of elements to return. If not supplied, the default is 100. For numbers larger than 100, each 100 items is counted as one request for pricing purposes. Maximum value is 100000.
- `sourceType` 'coin-api', required
- SourceCoinmarketcap
- `api_key` string, required — Your API Key. See here. The token is case sensitive.
- `data_type` 'latest' | 'historical', required — /latest: Latest market ticker quotes and averages for cryptocurrencies and exchanges. /historical: Intervals of historic market data like OHLCV data or data for use in charting libraries. See here.
- `symbols` string[] — Cryptocurrency symbols. (only used for quotes stream)
- `sourceType` 'coinmarketcap', required
- SourceConfigcat
- `username` string, required — Basic auth user name. See here.
- `password` string, required — Basic auth password. See here.
- `sourceType` 'configcat', required
- SourceConfluence
- `email` string, required — Your Confluence login email
- `api_token` string, required — Please follow the Jira confluence for generating an API token: generating an API token.
- `domain_name` string, required — Your Confluence domain name
- `sourceType` 'confluence', required
- SourceConvertkit
- `api_secret` string, required — API Secret
- `sourceType` 'convertkit', required
- SourceConvex
- `deployment_url` string, required
- `access_key` string, required — API access key used to retrieve data from Convex.
- `sourceType` 'convex', required
- SourceCopper
- `api_key` string, required — Copper API key
- `user_email` string, required — user email used to login in to Copper
- `sourceType` 'copper', required
- SourceCountercyclical
- `api_key` string, required
- `sourceType` 'countercyclical', required
- SourceCustomerIo
- `app_api_key` string, required
- `sourceType` 'customer-io', required
- SourceDatadog
- `api_key` string, required — Datadog API key
- `application_key` string, required — Datadog application key
- `query` string — The search query. This just applies to Incremental syncs. If empty, it'll collect all logs.
- `start_date` string — UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated. This just applies to Incremental syncs.
- `site` 'datadoghq.com' | 'us3.datadoghq.com' | 'us5.datadoghq.com' | 'datadoghq.eu' | 'ddog-gov.com' — The site where Datadog data resides in.
- `end_date` string — UTC date and time in the format 2017-01-25T00:00:00Z. Data after this date will not be replicated. An empty value will represent the current datetime for each execution. This just applies to Incremental syncs.
- `max_records_per_request` integer — Maximum number of records to collect per request.
- `queries` object[] — List of queries to be run and used as inputs.
- `name` string, required — The variable name for use in queries.
- `query` string, required — A classic query string.
- `data_source` 'metrics' | 'cloud_cost' | 'logs' | 'rum', required — A data source that is powered by the platform.
- `sourceType` 'datadog', required
- SourceDatascope
- `api_key` string, required — API Key
- `start_date` string, required — Start date for the data to be replicated
- `sourceType` 'datascope', required
- SourceDbt
- `api_key_2` string, required
- `account_id` string, required
- `sourceType` 'dbt', required
- SourceDelighted
- `api_key` string, required — A Delighted API key.
- `since` string, date-time, required — The date from which you'd like to replicate the data
- `sourceType` 'delighted', required
- SourceDixa
- `api_token` string, required — Dixa API token
- `batch_size` integer — Number of days to batch into one request. Max 31.
- `start_date` string, date-time, required — The connector pulls records updated from this date onwards.
- `sourceType` 'dixa', required
- SourceDockerhub
- `docker_username` string, required — Username of DockerHub person or organization (for https://hub.docker.com/v2/repositories/USERNAME/ API call)
- `sourceType` 'dockerhub', required
- SourceDremio
- `api_key` string, required — API Key that is generated when you authenticate to Dremio API
- `base_url` string, required — URL of your Dremio instance
- `sourceType` 'dremio', required
- SourceDrift
- `credentials` union
- object
- `credentials` 'oauth2.0'
- `client_id` string, required — The Client ID of your Drift developer application.
- `client_secret` string, required — The Client Secret of your Drift developer application.
- `access_token` string, required — Access Token for making authenticated requests.
- `refresh_token` string, required — Refresh Token to renew the expired Access Token.
- object
- `credentials` 'access_token'
- `access_token` string, required — Drift Access Token. See the docs for more information on how to generate this key.
- `email` string — Email used as parameter for contacts stream
- `sourceType` 'drift', required
- SourceDrip
- `api_key` string, required — API key to use. Find it at https://www.getdrip.com/user/edit
- `sourceType` 'drip', required
- SourceDropboxSign
- `api_key` string, required — API key to use. Find it at https://app.hellosign.com/home/myAccount#api
- `start_date` string, date-time, required
- `sourceType` 'dropbox-sign', required
- SourceDynamodb
- `credentials` object — Credentials for the service
- `auth_type` 'Role'
- `endpoint` string — the URL of the Dynamodb database
- `region` '' | 'af-south-1' | 'ap-east-1' | 'ap-northeast-1' | 'ap-northeast-2' | 'ap-northeast-3' | 'ap-south-1' | 'ap-south-2' | 'ap-southeast-1' | 'ap-southeast-2' | 'ap-southeast-3' | 'ap-southeast-4' | 'ca-central-1' | 'ca-west-1' | 'cn-north-1' | 'cn-northwest-1' | 'eu-central-1' | 'eu-central-2' | 'eu-north-1' | 'eu-south-1' | 'eu-south-2' | 'eu-west-1' | 'eu-west-2' | 'eu-west-3' | 'il-central-1' | 'me-central-1' | 'me-south-1' | 'sa-east-1' | 'us-east-1' | 'us-east-2' | 'us-gov-east-1' | 'us-gov-west-1' | 'us-west-1' | 'us-west-2' — The region of the Dynamodb database
- `reserved_attribute_names` string — Comma separated reserved attribute names present in your tables
- `ignore_missing_read_permissions_tables` boolean — Ignore tables with missing scan/read permissions
- `sourceType` 'dynamodb'
- SourceEasypost
- `username` string, required — The API Key from your easypost settings
- `start_date` string, date-time, required
- `sourceType` 'easypost', required
- SourceEmailoctopus
- `api_key` string, required — EmailOctopus API Key. See the docs for information on how to generate this key.
- `sourceType` 'emailoctopus', required
- SourceEmploymentHero
- `api_key` string, required
- `organization_configids` unknown[] — Organization ID which could be found as result of `organizations` stream to be used in other substreams
- unknown
- `employees_configids` unknown[] — Employees IDs in the given organisation found in `employees` stream for passing to sub-streams
- unknown
- `sourceType` 'employment-hero', required
- SourceEventbrite
- `private_token` string, required — The private token to use for authenticating API requests.
- `start_date` string, date-time, required
- `sourceType` 'eventbrite', required
- SourceExchangeRates
- `start_date` string, date, required — Start getting data from that date.
- `access_key` string, required — Your API Key. See here. The key is case sensitive.
- `base` string — ISO reference currency. See here. Free plan doesn't support Source Currency Switching, default base currency is EUR
- `ignore_weekends` boolean — Ignore weekends? (Exchanges don't run on weekends)
- `sourceType` 'exchange-rates', required
- SourceEzofficeinventory
- `api_key` string, required — Your EZOfficeInventory Access Token. API Access is disabled by default. Enable API Access in Settings > Integrations > API Integration and click on Update to generate a new access token
- `subdomain` string, required — The company name used in signup, also visible in the URL when logged in.
- `start_date` string, date-time, required — Earliest date you want to sync historical streams (inventory_histories, asset_histories, asset_stock_histories) from
- `sourceType` 'ezofficeinventory', required
- SourceFacebookMarketing
- `account_ids` string[], required — The Facebook Ad account ID(s) to pull data from. The Ad account ID number is in the account dropdown menu or in your browser's address bar of your Meta Ads Manager. See the docs for more information.
- `access_token` string — The value of the generated access token. From your App’s Dashboard, click on "Marketing API" then "Tools". Select permissions ads_management, ads_read, read_insights, business_management. Then click on "Get token". See the docs for more information.
- `credentials` union, required — Credentials for connecting to the Facebook Marketing API
- object
- `auth_type` 'Client', required
- `client_id` string, required — Client ID for the Facebook Marketing API
- `client_secret` string, required — Client Secret for the Facebook Marketing API
- `access_token` string — The value of the generated access token. From your App’s Dashboard, click on "Marketing API" then "Tools". Select permissions ads_management, ads_read, read_insights, business_management. Then click on "Get token". See the docs for more information.
- object
- `auth_type` 'Service', required
- `access_token` string, required — The value of the generated access token. From your App’s Dashboard, click on "Marketing API" then "Tools". Select permissions ads_management, ads_read, read_insights, business_management. Then click on "Get token". See the docs for more information.
- `start_date` string, date-time — The date from which you'd like to replicate data for all incremental streams, in the format YYYY-MM-DDT00:00:00Z. If not set then all data will be replicated for usual streams and only last 2 years for insight streams.
- `end_date` string, date-time — The date until which you'd like to replicate data for all incremental streams, in the format YYYY-MM-DDT00:00:00Z. All data generated between the start date and this end date will be replicated. Not setting this option will result in always syncing the latest data.
- `campaign_statuses` string[] — Select the statuses you want to be loaded in the stream. If no specific statuses are selected, the API's default behavior applies, and some statuses may be filtered out.
- `adset_statuses` string[] — Select the statuses you want to be loaded in the stream. If no specific statuses are selected, the API's default behavior applies, and some statuses may be filtered out.
- `ad_statuses` string[] — Select the statuses you want to be loaded in the stream. If no specific statuses are selected, the API's default behavior applies, and some statuses may be filtered out.
- `fetch_thumbnail_images` boolean — Set to active if you want to fetch the thumbnail_url and store the result in thumbnail_data_url for each Ad Creative.
- `custom_insights` object[] — A list which contains ad statistics entries, each entry must have a name and can contains fields, breakdowns or action_breakdowns. Click on "add" to fill this field.
- `name` string, required — The name value of insight
- `level` 'ad' | 'adset' | 'campaign' | 'account' — Chosen level for API
- `fields` string[] — A list of chosen fields for fields parameter
- `breakdowns` string[] — A list of chosen breakdowns for breakdowns
- `action_breakdowns` string[] — A list of chosen action_breakdowns for action_breakdowns
- `action_report_time` 'conversion' | 'impression' | 'mixed' — Determines the report time of action stats. For example, if a person saw the ad on Jan 1st but converted on Jan 2nd, when you query the API with action_report_time=impression, you see a conversion on Jan 1st. When you query the API with action_report_time=conversion, you see a conversion on Jan 2nd.
- `time_increment` integer — Time window in days by which to aggregate statistics. The sync will be chunked into N day intervals, where N is the number of days you specified. For example, if you set this value to 7, then all statistics will be reported as 7-day aggregates by starting from the start_date. If the start and end dates are October 1st and October 30th, then the connector will output 5 records: 01 - 06, 07 - 13, 14 - 20, 21 - 27, and 28 - 30 (3 days only). The minimum allowed value for this field is 1, and the maximum is 89.
- `start_date` string, date-time — The date from which you'd like to replicate data for this stream, in the format YYYY-MM-DDT00:00:00Z.
- `end_date` string, date-time — The date until which you'd like to replicate data for this stream, in the format YYYY-MM-DDT00:00:00Z. All data generated between the start date and this end date will be replicated. Not setting this option will result in always syncing the latest data.
- `insights_lookback_window` integer — The attribution window
- `insights_job_timeout` integer — The insights job timeout
- `page_size` integer — Page size used when sending requests to Facebook API to specify number of records per page when response has pagination. Most users do not need to set this field unless they specifically need to tune the connector to address specific issues or use cases.
- `insights_lookback_window` integer — The attribution window. Facebook freezes insight data 28 days after it was generated, which means that all data from the past 28 days may have changed since we last emitted it, so you can retrieve refreshed insights from the past by setting this parameter. If you set a custom lookback window value in Facebook account, please provide the same value here.
- `insights_job_timeout` integer — Insights Job Timeout establishes the maximum amount of time (in minutes) of waiting for the report job to complete. When timeout is reached the job is considered failed and we are trying to request smaller amount of data by breaking the job to few smaller ones. If you definitely know that 60 minutes is not enough for your report to be processed then you can decrease the timeout value, so we start breaking job to smaller parts faster.
- `sourceType` 'facebook-marketing', required
- SourceFactorial
- `api_key` string, required
- `start_date` string, date-time, required
- `limit` string — Max records per page limit
- `sourceType` 'factorial', required
- SourceFaker
- `count` integer — How many users should be generated in total. The purchases table will be scaled to match, with 10 purchases created per 10 users. This setting does not apply to the products stream.
- `seed` integer — Manually control the faker random seed to return the same values on subsequent runs (leave -1 for random)
- `records_per_slice` integer — How many fake records will be in each page (stream slice), before a state message is emitted?
- `always_updated` boolean — Should the updated_at values for every record be new each sync? Setting this to false will case the source to stop emitting records after COUNT records have been emitted.
- `parallelism` integer — How many parallel workers should we use to generate fake data? Choose a value equal to the number of CPUs you will allocate to this source.
- `sourceType` 'faker', required
- SourceFauna
- `domain` string, required — Domain of Fauna to query. Defaults db.fauna.com. See the docs.
- `port` integer, required — Endpoint port.
- `scheme` string, required — URL scheme.
- `secret` string, required — Fauna secret, used when authenticating with the database.
- `collection` object — Settings for the Fauna Collection.
- `page_size` integer, required — The page size used when reading documents from the database. The larger the page size, the faster the connector processes documents. However, if a page is too large, the connector may fail.
Choose your page size based on how large the documents are.
See the docs.
- `deletions` union, required — This only applies to incremental syncs.
Enabling deletion mode informs your destination of deleted documents.
Disabled - Leave this feature disabled, and ignore deleted documents.
Enabled - Enables this feature. When a document is deleted, the connector exports a record with a "deleted at" column containing the time that the document was deleted.
- object
- `deletion_mode` 'ignore', required
- object
- `deletion_mode` 'deleted_field', required
- `column` string, required — Name of the "deleted at" column.
- `sourceType` 'fauna', required
- SourceFile
- `dataset_name` string, required — The Name of the final table to replicate this file into (should include letters, numbers dash and underscores only).
- `format` 'csv' | 'json' | 'jsonl' | 'excel' | 'excel_binary' | 'fwf' | 'feather' | 'parquet' | 'yaml', required — The Format of the file which should be replicated (Warning: some formats may be experimental, please refer to the docs).
- `reader_options` string — This should be a string in JSON format. It depends on the chosen file format to provide additional options and tune its behavior.
- `url` string, required — The URL path to access the file which should be replicated.
- `provider` union, required — The storage Provider or Location of the file(s) which should be replicated.
- object
- `storage` 'HTTPS', required
- `user_agent` boolean — Add User-Agent to request
- object
- `storage` 'GCS', required
- `service_account_json` string — In order to access private Buckets stored on Google Cloud, this connector would need a service account json credentials with the proper permissions as described here. Please generate the credentials.json file and copy/paste its content to this field (expecting JSON formats). If accessing publicly available data, this field is not necessary.
- object
- `storage` 'S3', required
- `aws_access_key_id` string — In order to access private Buckets stored on AWS S3, this connector would need credentials with the proper permissions. If accessing publicly available data, this field is not necessary.
- `aws_secret_access_key` string — In order to access private Buckets stored on AWS S3, this connector would need credentials with the proper permissions. If accessing publicly available data, this field is not necessary.
- object
- `storage` 'AzBlob', required
- `storage_account` string, required — The globally unique name of the storage account that the desired blob sits within. See here for more details.
- `sas_token` string — To access Azure Blob Storage, this connector would need credentials with the proper permissions. One option is a SAS (Shared Access Signature) token. If accessing publicly available data, this field is not necessary.
- `shared_key` string — To access Azure Blob Storage, this connector would need credentials with the proper permissions. One option is a storage account shared key (aka account key or access key). If accessing publicly available data, this field is not necessary.
- object
- `storage` 'SSH', required
- `user` string, required
- `password` string
- `host` string, required
- `port` string
- object
- `storage` 'SCP', required
- `user` string, required
- `password` string
- `host` string, required
- `port` string
- object
- `storage` 'SFTP', required
- `user` string, required
- `password` string
- `host` string, required
- `port` string
- object
- `storage` 'local', required — WARNING: Note that the local storage URL available for reading must start with the local mount "/local/" at the moment until we implement more advanced docker mounting options.
- `sourceType` 'file', required
- SourceFirebolt
- `client_id` string, required — Firebolt service account ID.
- `client_secret` string, required — Firebolt secret, corresponding to the service account ID.
- `account` string, required — Firebolt account to login.
- `host` string — The host name of your Firebolt database.
- `database` string, required — The database to connect to.
- `engine` string, required — Engine name to connect to.
- `sourceType` 'firebolt', required
- SourceFleetio
- `api_key` string, required
- `account_token` string, required
- `sourceType` 'fleetio', required
- SourceFlexport
- `api_key` string, required
- `start_date` string, date-time, required
- `sourceType` 'flexport', required
- SourceFreeAgentConnector
- `client_id` string, required
- `client_secret` string, required
- `client_refresh_token_2` string, required
- `updated_since` string, date-time
- `payroll_year` number
- `sourceType` 'free-agent-connector', required
- SourceFreshcaller
- `domain` string, required — Used to construct Base URL for the Freshcaller APIs
- `api_key` string, required — Freshcaller API Key. See the docs for more information on how to obtain this key.
- `requests_per_minute` integer — The number of requests per minute that this source allowed to use. There is a rate limit of 50 requests per minute per app per account.
- `start_date` string, date-time — UTC date and time. Any data created after this date will be replicated.
- `sync_lag_minutes` integer — Lag in minutes for each sync, i.e., at time T, data for the time range [prev_sync_time, T-30] will be fetched
- `sourceType` 'freshcaller', required
- SourceFreshchat
- `api_key` string, required
- `start_date` string, date-time, required
- `account_name` string, required — The unique account name for your Freshchat instance
- `sourceType` 'freshchat', required
- SourceFreshdesk
- `api_key` string, required — Freshdesk API Key. See the docs for more information on how to obtain this key.
- `domain` string, required — Freshdesk domain
- `requests_per_minute` integer — The number of requests per minute that this source allowed to use. There is a rate limit of 50 requests per minute per app per account.
- `start_date` string, date-time — UTC date and time. Any data created after this date will be replicated. If this parameter is not set, all data will be replicated.
- `lookback_window_in_days` integer — Number of days for lookback window for the stream Satisfaction Ratings
- `sourceType` 'freshdesk', required
- SourceFreshsales
- `api_key` string, required — Freshsales API Key. See here. The key is case sensitive.
- `domain_name` string, required — The Name of your Freshsales domain
- `sourceType` 'freshsales', required
- SourceFreshservice
- `api_key` string, required — Freshservice API Key. See here. The key is case sensitive.
- `domain_name` string, required — The name of your Freshservice domain
- `start_date` string, date-time, required — UTC date and time in the format 2020-10-01T00:00:00Z. Any data before this date will not be replicated.
- `sourceType` 'freshservice', required
- SourceFront
- `api_key` string, required
- `start_date` string, date-time, required
- `page_limit` string — Page limit for the responses
- `sourceType` 'front', required
- SourceGainsightPx
- `api_key` string, required — The Aptrinsic API Key which is recieved from the dashboard settings (ref - https://app.aptrinsic.com/settings/api-keys)
- `sourceType` 'gainsight-px', required
- SourceGcs — NOTE: When this Spec is changed, legacy_config_transformer.py must also be modified to uptake the changes because it is responsible for converting legacy GCS configs into file based configs using the File-Based CDK.
- `start_date` string, date-time — UTC date and time in the format 2017-01-25T00:00:00.000000Z. Any file modified before this date will not be replicated.
- `streams` object[], required — Each instance of this configuration defines a stream. Use this to define which files belong in the stream, their format, and how they should be parsed and validated. When sending data to warehouse destination such as Snowflake or BigQuery, each stream is a separate table.
- `name` string, required — The name of the stream.
- `globs` string[] — The pattern used to specify which files should be selected from the file system. For more information on glob pattern matching look here.
- `validation_policy` 'Emit Record' | 'Skip Record' | 'Wait for Discover' — The name of the validation policy that dictates sync behavior when a record does not adhere to the stream schema.
- `input_schema` string — The schema that will be used to validate records extracted from the file. This will override the stream schema that is auto-detected from incoming files.
- `days_to_sync_if_history_is_full` integer — When the state history of the file store is full, syncs will only read files that were last modified in the provided day range.
- `format` union, required — The configuration options that are used to alter how to read incoming files that deviate from the standard formatting.
- object
- `filetype` 'avro', required
- `double_as_string` boolean — Whether to convert double fields to strings. This is recommended if you have decimal numbers with a high degree of precision because there can be a loss precision when handling floating point numbers.
- object
- `filetype` 'csv', required
- `delimiter` string — The character delimiting individual cells in the CSV data. This may only be a 1-character string. For tab-delimited data enter '\t'.
- `quote_char` string — The character used for quoting CSV values. To disallow quoting, make this field blank.
- `escape_char` string — The character used for escaping special characters. To disallow escaping, leave this field blank.
- `encoding` string — The character encoding of the CSV data. Leave blank to default to UTF8. See list of python encodings for allowable options.
- `double_quote` boolean — Whether two quotes in a quoted CSV value denote a single quote in the data.
- `null_values` string[] — A set of case-sensitive strings that should be interpreted as null values. For example, if the value 'NA' should be interpreted as null, enter 'NA' in this field.
- `strings_can_be_null` boolean — Whether strings can be interpreted as null values. If true, strings that match the null_values set will be interpreted as null. If false, strings that match the null_values set will be interpreted as the string itself.
- `skip_rows_before_header` integer — The number of rows to skip before the header row. For example, if the header row is on the 3rd row, enter 2 in this field.
- `skip_rows_after_header` integer — The number of rows to skip after the header row.
- `header_definition` union — How headers will be defined. `User Provided` assumes the CSV does not have a header row and uses the headers provided and `Autogenerated` assumes the CSV does not have a header row and the CDK will generate headers using for `f{i}` where `i` is the index starting from 0. Else, the default behavior is to use the header from the CSV file. If a user wants to autogenerate or provide column names for a CSV having headers, they can skip rows.
- …
- `true_values` string[] — A set of case-sensitive strings that should be interpreted as true values.
- `false_values` string[] — A set of case-sensitive strings that should be interpreted as false values.
- `ignore_errors_on_fields_mismatch` boolean — Whether to ignore errors that occur when the number of fields in the CSV does not match the number of columns in the schema.
- object
- `filetype` 'jsonl', required
- object
- `filetype` 'parquet', required
- `decimal_as_float` boolean — Whether to convert decimal fields to floats. There is a loss of precision when converting decimals to floats, so this is not recommended.
- object — Extract text from document formats (.pdf, .docx, .md, .pptx) and emit as one record per file.
- `filetype` 'unstructured', required
- `skip_unprocessable_files` boolean — If true, skip files that cannot be parsed and pass the error message along as the _ab_source_file_parse_error field. If false, fail the sync.
- `strategy` 'auto' | 'fast' | 'ocr_only' | 'hi_res' — The strategy used to parse documents. `fast` extracts text directly from the document which doesn't work for all files. `ocr_only` is more reliable, but slower. `hi_res` is the most reliable, but requires an API key and a hosted instance of unstructured and can't be used with local mode. See the unstructured.io documentation for more details: https://unstructured-io.github.io/unstructured/core/partition.html#partition-pdf
- `processing` union — Processing configuration
- …
- object
- `filetype` 'excel', required
- `schemaless` boolean — When enabled, syncs will not validate or structure records against the stream's schema.
- `recent_n_files_to_read_for_schema_discovery` integer — The number of resent files which will be used to discover the schema for this stream.
- `service_account` string, required — Enter your Google Cloud service account key in JSON format
- `bucket` string, required — Name of the GCS bucket where the file(s) exist.
- `sourceType` 'gcs', required
- SourceGetlago
- `api_url` string — Your Lago API URL
- `api_key` string, required — Your API Key. See here.
- `sourceType` 'getlago', required
- SourceGithub
- `credentials` union, required — Choose how to authenticate to GitHub
- object
- `option_title` 'OAuth Credentials'
- `access_token` string, required — OAuth access token
- `client_id` string — OAuth Client Id
- `client_secret` string — OAuth Client secret
- object
- `option_title` 'PAT Credentials'
- `personal_access_token` string, required — Log into GitHub and then generate a personal access token. To load balance your API quota consumption across multiple API tokens, input multiple tokens separated with ","
- `repositories` string[], required — List of GitHub organizations/repositories, e.g. `airbytehq/airbyte` for single repository, `airbytehq/*` for get all repositories from organization and `airbytehq/a* for matching multiple repositories by pattern.
- `start_date` string, date-time — The date from which you'd like to replicate data from GitHub in the format YYYY-MM-DDT00:00:00Z. If the date is not set, all data will be replicated. For the streams which support this configuration, only data generated on or after the start date will be replicated. This field doesn't apply to all streams, see the docs for more info
- `api_url` string — Please enter your basic URL from self-hosted GitHub instance or leave it empty to use GitHub.
- `branches` string[] — List of GitHub repository branches to pull commits for, e.g. `airbytehq/airbyte/master`. If no branches are specified for a repository, the default branch will be pulled.
- `max_waiting_time` integer — Max Waiting Time for rate limit. Set higher value to wait till rate limits will be resetted to continue sync
- `sourceType` 'github', required
- SourceGitlab
- `credentials` union, required
- object
- `auth_type` 'oauth2.0'
- `client_id` string, required — The API ID of the Gitlab developer application.
- `client_secret` string, required — The API Secret the Gitlab developer application.
- `access_token` string, required — Access Token for making authenticated requests.
- `token_expiry_date` string, date-time, required — The date-time when the access token should be refreshed.
- `refresh_token` string, required — The key to refresh the expired access_token.
- object
- `auth_type` 'access_token'
- `access_token` string, required — Log into your Gitlab account and then generate a personal Access Token.
- `start_date` string, date-time — The date from which you'd like to replicate data for GitLab API, in the format YYYY-MM-DDT00:00:00Z. Optional. If not set, all data will be replicated. All data generated after this date will be replicated.
- `api_url` string — Please enter your basic URL from GitLab instance.
- `groups_list` string[] — List of groups. e.g. airbyte.io.
- `projects_list` string[] — Space-delimited list of projects. e.g. airbyte.io/documentation meltano/tap-gitlab.
- `sourceType` 'gitlab', required
- SourceGlassfrog
- `api_key` string, required — API key provided by Glassfrog
- `sourceType` 'glassfrog', required
- SourceGmail
- `client_id` string, required
- `client_secret` string, required
- `client_refresh_token` string, required
- `include_spam_and_trash` boolean — Include drafts/messages from SPAM and TRASH in the results. Defaults to false.
- `sourceType` 'gmail', required
- SourceGnews
- `api_key` string, required — API Key
- `query` string, required — This parameter allows you to specify your search keywords to find the news articles you are looking for. The keywords will be used to return the most relevant articles. It is possible to use logical operators with keywords. - Phrase Search Operator: This operator allows you to make an exact search. Keywords surrounded by quotation marks are used to search for articles with the exact same keyword sequence. For example the query: "Apple iPhone" will return articles matching at least once this sequence of keywords. - Logical AND Operator: This operator allows you to make sure that several keywords are all used in the article search. By default the space character acts as an AND operator, it is possible to replace the space character by AND to obtain the same result. For example the query: Apple Microsoft is equivalent to Apple AND Microsoft - Logical OR Operator: This operator allows you to retrieve articles matching the keyword a or the keyword b. It is important to note that this operator has a higher precedence than the AND operator. For example the query: Apple OR Microsoft will return all articles matching the keyword Apple as well as all articles matching the keyword Microsoft - Logical NOT Operator: This operator allows you to remove from the results the articles corresponding to the specified keywords. To use it, you need to add NOT in front of each word or phrase surrounded by quotes. For example the query: Apple NOT iPhone will return all articles matching the keyword Apple but not the keyword iPhone
- `language` 'ar' | 'zh' | 'nl' | 'en' | 'fr' | 'de' | 'el' | 'he' | 'hi' | 'it' | 'ja' | 'ml' | 'mr' | 'no' | 'pt' | 'ro' | 'ru' | 'es' | 'sv' | 'ta' | 'te' | 'uk'
- `country` 'au' | 'br' | 'ca' | 'cn' | 'eg' | 'fr' | 'de' | 'gr' | 'hk' | 'in' | 'ie' | 'il' | 'it' | 'jp' | 'nl' | 'no' | 'pk' | 'pe' | 'ph' | 'pt' | 'ro' | 'ru' | 'sg' | 'es' | 'se' | 'ch' | 'tw' | 'ua' | 'gb' | 'us' — This parameter allows you to specify the country where the news articles returned by the API were published, the contents of the articles are not necessarily related to the specified country. You have to set as value the 2 letters code of the country you want to filter.
- `in` string[] — This parameter allows you to choose in which attributes the keywords are searched. The attributes that can be set are title, description and content. It is possible to combine several attributes.
- `nullable` string[] — This parameter allows you to specify the attributes that you allow to return null values. The attributes that can be set are title, description and content. It is possible to combine several attributes
- `start_date` string — This parameter allows you to filter the articles that have a publication date greater than or equal to the specified value. The date must respect the following format: YYYY-MM-DD hh:mm:ss (in UTC)
- `end_date` string — This parameter allows you to filter the articles that have a publication date smaller than or equal to the specified value. The date must respect the following format: YYYY-MM-DD hh:mm:ss (in UTC)
- `sortby` 'publishedAt' | 'relevance' — This parameter allows you to choose with which type of sorting the articles should be returned. Two values are possible: - publishedAt = sort by publication date, the articles with the most recent publication date are returned first - relevance = sort by best match to keywords, the articles with the best match are returned first
- `top_headlines_query` string — This parameter allows you to specify your search keywords to find the news articles you are looking for. The keywords will be used to return the most relevant articles. It is possible to use logical operators with keywords. - Phrase Search Operator: This operator allows you to make an exact search. Keywords surrounded by quotation marks are used to search for articles with the exact same keyword sequence. For example the query: "Apple iPhone" will return articles matching at least once this sequence of keywords. - Logical AND Operator: This operator allows you to make sure that several keywords are all used in the article search. By default the space character acts as an AND operator, it is possible to replace the space character by AND to obtain the same result. For example the query: Apple Microsoft is equivalent to Apple AND Microsoft - Logical OR Operator: This operator allows you to retrieve articles matching the keyword a or the keyword b. It is important to note that this operator has a higher precedence than the AND operator. For example the query: Apple OR Microsoft will return all articles matching the keyword Apple as well as all articles matching the keyword Microsoft - Logical NOT Operator: This operator allows you to remove from the results the articles corresponding to the specified keywords. To use it, you need to add NOT in front of each word or phrase surrounded by quotes. For example the query: Apple NOT iPhone will return all articles matching the keyword Apple but not the keyword iPhone
- `top_headlines_topic` 'breaking-news' | 'world' | 'nation' | 'business' | 'technology' | 'entertainment' | 'sports' | 'science' | 'health' — This parameter allows you to change the category for the request.
- `sourceType` 'gnews', required
- SourceGocardless
- `access_token` string, required — Gocardless API TOKEN
- `gocardless_environment` 'sandbox' | 'live', required — Environment you are trying to connect to.
- `gocardless_version` string, required — GoCardless version. This is a date. You can find the latest here: https://developer.gocardless.com/api-reference/#api-usage-making-requests
- `start_date` string, required — UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.
- `sourceType` 'gocardless', required
- SourceGoldcast
- `access_key` string, required — Your API Access Key. See here. The key is case sensitive.
- `sourceType` 'goldcast', required
- SourceGong
- `access_key` string, required — Gong Access Key
- `access_key_secret` string, required — Gong Access Key Secret
- `start_date` string — The date from which to list calls, in the ISO-8601 format; if not specified, the calls start with the earliest recorded call. For web-conference calls recorded by Gong, the date denotes its scheduled time, otherwise, it denotes its actual start time.
- `sourceType` 'gong', required
- SourceGoogleAds
- `credentials` object, required
- `developer_token` string, required — The Developer Token granted by Google to use their APIs. For detailed instructions on finding this value, refer to our documentation.
- `client_id` string, required — The Client ID of your Google Ads developer application. For detailed instructions on finding this value, refer to our documentation.
- `client_secret` string, required — The Client Secret of your Google Ads developer application. For detailed instructions on finding this value, refer to our documentation.
- `refresh_token` string, required — The token used to obtain a new Access Token. For detailed instructions on finding this value, refer to our documentation.
- `access_token` string — The Access Token for making authenticated requests. For detailed instructions on finding this value, refer to our documentation.
- `customer_id` string — Comma-separated list of (client) customer IDs. Each customer ID must be specified as a 10-digit number without dashes. For detailed instructions on finding this value, refer to our documentation.
- `customer_status_filter` string[] — A list of customer statuses to filter on. For detailed info about what each status mean refer to Google Ads documentation.
- `start_date` string, date — UTC date in the format YYYY-MM-DD. Any data before this date will not be replicated. (Default value of two years ago is used if not set)
- `end_date` string, date — UTC date in the format YYYY-MM-DD. Any data after this date will not be replicated. (Default value of today is used if not set)
- `custom_queries_array` object[]
- `query` string, required — A custom defined GAQL query for building the report. Avoid including the segments.date field; wherever possible, Airbyte will automatically include it for incremental syncs. For more information, refer to Google's documentation.
- `table_name` string, required — The table name in your destination database for the chosen query.
- `conversion_window_days` integer — A conversion window is the number of days after an ad interaction (such as an ad click or video view) during which a conversion, such as a purchase, is recorded in Google Ads. For more information, see Google's documentation.
- `sourceType` 'google-ads', required
- SourceGoogleAnalyticsDataApi
- `credentials` union — Credentials for the service
- object
- `auth_type` 'Client'
- `client_id` string, required — The Client ID of your Google Analytics developer application.
- `client_secret` string, required — The Client Secret of your Google Analytics developer application.
- `refresh_token` string, required — The token for obtaining a new access token.
- `access_token` string — Access Token for making authenticated requests.
- object
- `auth_type` 'Service'
- `credentials_json` string, required — The JSON key linked to the service account used for authorization. For steps on obtaining this key, refer to the setup guide.
- `property_ids` string[], required — A list of your Property IDs. The Property ID is a unique number assigned to each property in Google Analytics, found in your GA4 property URL. This ID allows the connector to track the specific events associated with your property. Refer to the Google Analytics documentation to locate your property ID.
- `date_ranges_start_date` string, date — The start date from which to replicate report data in the format YYYY-MM-DD. Data generated before this date will not be included in the report. Not applied to custom Cohort reports.
- `custom_reports_array` object[] — You can add your Custom Analytics report by creating one.
- `name` string, required — The name of the custom report, this name would be used as stream name.
- `dimensions` string[], required — A list of dimensions.
- `metrics` string[], required — A list of metrics.
- `dimensionFilter` union — Dimensions filter
- object — The FilterExpressions in andGroup have an AND relationship.
- `filter_type` 'andGroup', required
- `expressions` object[], required
- …
- object — The FilterExpressions in orGroup have an OR relationship.
- `filter_type` 'orGroup', required
- `expressions` object[], required
- …
- object — The FilterExpression is NOT of notExpression.
- `filter_type` 'notExpression'
- `expression` object
- …
- object — A primitive filter. In the same FilterExpression, all of the filter's field names need to be either all dimensions.
- `filter_type` 'filter'
- `field_name` string, required
- `filter` union, required
- …
- `metricFilter` union — Metrics filter
- object — The FilterExpressions in andGroup have an AND relationship.
- `filter_type` 'andGroup', required
- `expressions` object[], required
- …
- object — The FilterExpressions in orGroup have an OR relationship.
- `filter_type` 'orGroup', required
- `expressions` object[], required
- …
- object — The FilterExpression is NOT of notExpression.
- `filter_type` 'notExpression'
- `expression` object
- …
- object — A primitive filter. In the same FilterExpression, all of the filter's field names need to be either all metrics.
- `filter_type` 'filter'
- `field_name` string, required
- `filter` union, required
- …
- `cohortSpec` union — Cohort reports creates a time series of user retention for the cohort.
- object
- `enabled` 'false'
- object
- `enabled` 'true'
- `cohorts` object[]
- …
- `cohortsRange` object
- …
- `cohortReportSettings` object — Optional settings for a cohort report.
- …
- `window_in_days` integer — The interval in days for each data request made to the Google Analytics API. A larger value speeds up data sync, but increases the chance of data sampling, which may result in inaccuracies. We recommend a value of 1 to minimize sampling, unless speed is an absolute priority over accuracy. Acceptable values range from 1 to 364. Does not apply to custom Cohort reports. More information is available in the documentation.
- `lookback_window` integer — Since attribution changes after the event date, and Google Analytics has a data processing latency, we should specify how many days in the past we should refresh the data in every run. So if you set it at 5 days, in every sync it will fetch the last bookmark date minus 5 days.
- `keep_empty_rows` boolean — If false, each row with all metrics equal to 0 will not be returned. If true, these rows will be returned if they are not separately removed by a filter. More information is available in the documentation.
- `convert_conversions_event` boolean — Enables conversion of `conversions:*` event metrics from integers to floats. This is beneficial for preventing data rounding when the API returns float values for any `conversions:*` fields.
- `sourceType` 'google-analytics-data-api', required
- SourceGoogleCalendar
- `client_id` string, required
- `client_secret` string, required
- `client_refresh_token_2` string, required
- `calendarid` string, required
- `sourceType` 'google-calendar', required
- SourceGoogleDirectory
- `credentials` union — Google APIs use the OAuth 2.0 protocol for authentication and authorization. The Source supports Web server application and Service accounts scenarios.
- object — For these scenario user only needs to give permission to read Google Directory data.
- `credentials_title` 'Web server app' — Authentication Scenario
- `client_id` string, required — The Client ID of the developer application.
- `client_secret` string, required — The Client Secret of the developer application.
- `refresh_token` string, required — The Token for obtaining a new access token.
- object — For these scenario user should obtain service account's credentials from the Google API Console and provide delegated email.
- `credentials_title` 'Service accounts' — Authentication Scenario
- `credentials_json` string, required — The contents of the JSON service account key. See the docs for more information on how to generate this key.
- `email` string, required — The email of the user, which has permissions to access the Google Workspace Admin APIs.
- `sourceType` 'google-directory', required
- SourceGoogleDrive — Used during spec; allows the developer to configure the cloud provider specific options that are needed when users configure a file-based source.
- `start_date` string, date-time — UTC date and time in the format 2017-01-25T00:00:00.000000Z. Any file modified before this date will not be replicated.
- `streams` object[], required — Each instance of this configuration defines a stream. Use this to define which files belong in the stream, their format, and how they should be parsed and validated. When sending data to warehouse destination such as Snowflake or BigQuery, each stream is a separate table.
- `name` string, required — The name of the stream.
- `globs` string[] — The pattern used to specify which files should be selected from the file system. For more information on glob pattern matching look here.
- `validation_policy` 'Emit Record' | 'Skip Record' | 'Wait for Discover' — The name of the validation policy that dictates sync behavior when a record does not adhere to the stream schema.
- `input_schema` string — The schema that will be used to validate records extracted from the file. This will override the stream schema that is auto-detected from incoming files.
- `days_to_sync_if_history_is_full` integer — When the state history of the file store is full, syncs will only read files that were last modified in the provided day range.
- `format` union, required — The configuration options that are used to alter how to read incoming files that deviate from the standard formatting.
- object
- `filetype` 'avro', required
- `double_as_string` boolean — Whether to convert double fields to strings. This is recommended if you have decimal numbers with a high degree of precision because there can be a loss precision when handling floating point numbers.
- object
- `filetype` 'csv', required
- `delimiter` string — The character delimiting individual cells in the CSV data. This may only be a 1-character string. For tab-delimited data enter '\t'.
- `quote_char` string — The character used for quoting CSV values. To disallow quoting, make this field blank.
- `escape_char` string — The character used for escaping special characters. To disallow escaping, leave this field blank.
- `encoding` string — The character encoding of the CSV data. Leave blank to default to UTF8. See list of python encodings for allowable options.
- `double_quote` boolean — Whether two quotes in a quoted CSV value denote a single quote in the data.
- `null_values` string[] — A set of case-sensitive strings that should be interpreted as null values. For example, if the value 'NA' should be interpreted as null, enter 'NA' in this field.
- `strings_can_be_null` boolean — Whether strings can be interpreted as null values. If true, strings that match the null_values set will be interpreted as null. If false, strings that match the null_values set will be interpreted as the string itself.
- `skip_rows_before_header` integer — The number of rows to skip before the header row. For example, if the header row is on the 3rd row, enter 2 in this field.
- `skip_rows_after_header` integer — The number of rows to skip after the header row.
- `header_definition` union — How headers will be defined. `User Provided` assumes the CSV does not have a header row and uses the headers provided and `Autogenerated` assumes the CSV does not have a header row and the CDK will generate headers using for `f{i}` where `i` is the index starting from 0. Else, the default behavior is to use the header from the CSV file. If a user wants to autogenerate or provide column names for a CSV having headers, they can skip rows.
- …
- `true_values` string[] — A set of case-sensitive strings that should be interpreted as true values.
- `false_values` string[] — A set of case-sensitive strings that should be interpreted as false values.
- `ignore_errors_on_fields_mismatch` boolean — Whether to ignore errors that occur when the number of fields in the CSV does not match the number of columns in the schema.
- object
- `filetype` 'jsonl', required
- object
- `filetype` 'parquet', required
- `decimal_as_float` boolean — Whether to convert decimal fields to floats. There is a loss of precision when converting decimals to floats, so this is not recommended.
- object — Extract text from document formats (.pdf, .docx, .md, .pptx) and emit as one record per file.
- `filetype` 'unstructured', required
- `skip_unprocessable_files` boolean — If true, skip files that cannot be parsed and pass the error message along as the _ab_source_file_parse_error field. If false, fail the sync.
- `strategy` 'auto' | 'fast' | 'ocr_only' | 'hi_res' — The strategy used to parse documents. `fast` extracts text directly from the document which doesn't work for all files. `ocr_only` is more reliable, but slower. `hi_res` is the most reliable, but requires an API key and a hosted instance of unstructured and can't be used with local mode. See the unstructured.io documentation for more details: https://unstructured-io.github.io/unstructured/core/partition.html#partition-pdf
- `processing` object — Process files locally, supporting `fast` and `ocr` modes. This is the default option.
- …
- `schemaless` boolean — When enabled, syncs will not validate or structure records against the stream's schema.
- `folder_url` string, required — URL for the folder you want to sync. Using individual streams and glob patterns, it's possible to only sync a subset of all files located in the folder.
- `credentials` union, required — Credentials for connecting to the Google Drive API
- object
- `auth_type` 'Client', required
- `client_id` string, required — Client ID for the Google Drive API
- `client_secret` string, required — Client Secret for the Google Drive API
- `refresh_token` string, required — Refresh Token for the Google Drive API
- object
- `auth_type` 'Service', required
- `service_account_info` string, required — The JSON key of the service account to use for authorization. Read more here.
- `sourceType` 'google-drive', required
- SourceGooglePagespeedInsights
- `api_key` string — Google PageSpeed API Key. See here. The key is optional - however the API is heavily rate limited when using without API Key. Creating and using the API key therefore is recommended. The key is case sensitive.
- `categories` string[], required — Defines which Lighthouse category to run. One or many of: "accessibility", "best-practices", "performance", "pwa", "seo".
- `strategies` string[], required — The analyses strategy to use. Either "desktop" or "mobile".
- `urls` string[], required — The URLs to retrieve pagespeed information from. The connector will attempt to sync PageSpeed reports for all the defined URLs. Format: https://(www.)url.domain
- `sourceType` 'google-pagespeed-insights', required
- SourceGoogleSearchConsole
- `site_urls` string[], required — The URLs of the website property attached to your GSC account. Learn more about properties here.
- `start_date` string, date — UTC date in the format YYYY-MM-DD. Any data before this date will not be replicated.
- `end_date` string, date — UTC date in the format YYYY-MM-DD. Any data created after this date will not be replicated. Must be greater or equal to the start date field. Leaving this field blank will replicate all data from the start date onward.
- `authorization` union, required
- object
- `auth_type` 'Client', required
- `client_id` string, required — The client ID of your Google Search Console developer application. Read more here.
- `client_secret` string, required — The client secret of your Google Search Console developer application. Read more here.
- `access_token` string — Access token for making authenticated requests. Read more here.
- `refresh_token` string, required — The token for obtaining a new access token. Read more here.
- object
- `auth_type` 'Service', required
- `service_account_info` string, required — The JSON key of the service account to use for authorization. Read more here.
- `email` string, required — The email of the user which has permissions to access the Google Workspace Admin APIs.
- `custom_reports_array` object[] — You can add your Custom Analytics report by creating one.
- `name` string, required — The name of the custom report, this name would be used as stream name
- `dimensions` string[], required — A list of available dimensions. Please note, that for technical reasons `date` is the default dimension which will be included in your query whether you specify it or not. Primary key will consist of your custom dimensions and the default dimension along with `site_url` and `search_type`.
- `data_state` 'final' | 'all' — If set to 'final', the returned data will include only finalized, stable data. If set to 'all', fresh data will be included. When using Incremental sync mode, we do not recommend setting this parameter to 'all' as it may cause data loss. More information can be found in our full documentation.
- `sourceType` 'google-search-console', required
- SourceGoogleSheets
- `batch_size` integer — Default value is 200. An integer representing row batch size for each sent request to Google Sheets API. Row batch size means how many rows are processed from the google sheet, for example default value 200 would process rows 1-201, then 201-401 and so on. Based on Google Sheets API limits documentation, it is possible to send up to 300 requests per minute, but each individual request has to be processed under 180 seconds, otherwise the request returns a timeout error. In regards to this information, consider network speed and number of columns of the google sheet when deciding a batch_size value. Default value should cover most of the cases, but if a google sheet has over 100,000 records or more, consider increasing batch_size value.
- `spreadsheet_id` string, required — Enter the link to the Google spreadsheet you want to sync. To copy the link, click the 'Share' button in the top-right corner of the spreadsheet, then click 'Copy link'.
- `names_conversion` boolean — Enables the conversion of column names to a standardized, SQL-compliant format. For example, 'My Name' -> 'my_name'. Enable this option if your destination is SQL-based.
- `credentials` union, required — Credentials for connecting to the Google Sheets API
- object
- `auth_type` 'Client', required
- `client_id` string, required — Enter your Google application's Client ID. See Google's documentation for more information.
- `client_secret` string, required — Enter your Google application's Client Secret. See Google's documentation for more information.
- `refresh_token` string, required — Enter your Google application's refresh token. See Google's documentation for more information.
- object
- `auth_type` 'Service', required
- `service_account_info` string, required — The JSON key of the service account to use for authorization. Read more here.
- `sourceType` 'google-sheets', required
- SourceGoogleTasks
- `api_key` string, required
- `start_date` string, date-time, required
- `records_limit` string — The maximum number of records to be returned per request
- `sourceType` 'google-tasks', required
- SourceGoogleWebfonts
- `api_key` string, required — API key is required to access google apis, For getting your's goto google console and generate api key for Webfonts
- `alt` string — Optional, Available params- json, media, proto
- `prettyPrint` string — Optional, boolean type
- `sort` string — Optional, to find how to sort
- `sourceType` 'google-webfonts', required
- SourceGorgias
- `username` string, required
- `password` string
- `domain_name` string, required — Domain name given for gorgias, found as your url prefix for accessing your website
- `start_date` string, date-time, required
- `sourceType` 'gorgias', required
- SourceGreenhouse
- `api_key` string, required — Greenhouse API Key. See the docs for more information on how to generate this key.
- `sourceType` 'greenhouse', required
- SourceGridly
- `api_key` string, required
- `grid_id` string, required — ID of a grid, or can be ID of a branch
- `sourceType` 'gridly', required
- SourceGuru
- `username` string, required
- `password` string
- `start_date` string, date-time, required
- `team_id` string — Team ID received through response of /teams streams, make sure about access to the team
- `search_cards_query` string — Query for searching cards
- `sourceType` 'guru', required
- SourceGutendex
- `author_year_end` string — (Optional) Defines the maximum birth year of the authors. Books by authors born after the end year will not be returned. Supports both positive (CE) or negative (BCE) integer values
- `author_year_start` string — (Optional) Defines the minimum birth year of the authors. Books by authors born prior to the start year will not be returned. Supports both positive (CE) or negative (BCE) integer values
- `copyright` string — (Optional) Use this to find books with a certain copyright status - true for books with existing copyrights, false for books in the public domain in the USA, or null for books with no available copyright information.
- `languages` string — (Optional) Use this to find books in any of a list of languages. They must be comma-separated, two-character language codes.
- `search` string — (Optional) Use this to search author names and book titles with given words. They must be separated by a space (i.e. %20 in URL-encoded format) and are case-insensitive.
- `sort` string — (Optional) Use this to sort books - ascending for Project Gutenberg ID numbers from lowest to highest, descending for IDs highest to lowest, or popular (the default) for most popular to least popular by number of downloads.
- `topic` string — (Optional) Use this to search for a case-insensitive key-phrase in books' bookshelves or subjects.
- `sourceType` 'gutendex', required
- SourceHardcodedRecords
- `count` integer — How many records per stream should be generated
- `sourceType` 'hardcoded-records', required
- SourceHarvest
- `account_id` string, required — Harvest account ID. Required for all Harvest requests in pair with Personal Access Token
- `replication_start_date` string, date-time, required — UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.
- `credentials` union — Choose how to authenticate to Harvest.
- object
- `auth_type` 'Client'
- `client_id` string, required — The Client ID of your Harvest developer application.
- `client_secret` string, required — The Client Secret of your Harvest developer application.
- `refresh_token` string, required — Refresh Token to renew the expired Access Token.
- object
- `api_token` string, required — Log into Harvest and then create new personal access token.
- `auth_type` 'Token'
- `sourceType` 'harvest', required
- SourceHeight
- `api_key` string, required
- `start_date` string, date-time, required
- `search_query` string — Search query to be used with search stream
- `sourceType` 'height', required
- SourceHibob
- `username` string, required
- `password` string
- `is_sandbox` boolean, required — Toggle true if this instance is a HiBob sandbox
- `sourceType` 'hibob', required
- SourceHighLevel
- `location_id` string, required
- `api_key` string, required
- `start_date` string, date-time, required
- `sourceType` 'high-level', required
- SourceHubplanner
- `api_key` string, required — Hubplanner API key. See https://github.com/hubplanner/API#authentication for more details.
- `sourceType` 'hubplanner', required
- SourceHubspot
- `start_date` string, date-time — UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated. If not set, "2006-06-01T00:00:00Z" (Hubspot creation date) will be used as start date. It's recommended to provide relevant to your data start date value to optimize synchronization.
- `credentials` union, required — Choose how to authenticate to HubSpot.
- object
- `credentials_title` 'OAuth Credentials', required — Name of the credentials
- `client_id` string, required — The Client ID of your HubSpot developer application. See the Hubspot docs if you need help finding this ID.
- `client_secret` string, required — The client secret for your HubSpot developer application. See the Hubspot docs if you need help finding this secret.
- `refresh_token` string, required — Refresh token to renew an expired access token. See the Hubspot docs if you need help finding this token.
- object
- `credentials_title` 'Private App Credentials', required — Name of the credentials set
- `access_token` string, required — HubSpot Access token. See the Hubspot docs if you need help finding this token.
- `enable_experimental_streams` boolean — If enabled then experimental streams become available for sync.
- `sourceType` 'hubspot', required
- SourceIlluminaBasespace
- `access_token` string, required — BaseSpace access token. Instructions for obtaining your access token can be found in the BaseSpace Developer Documentation.
- `domain` string, required — Domain name of the BaseSpace instance (e.g., euw2.sh.basespace.illumina.com)
- `user` string, required — Providing a user ID restricts the returned data to what that user can access. If you use the default ('current'), all data accessible to the user associated with the API key will be shown.
- `sourceType` 'illumina-basespace', required
- SourceIncidentIo
- `api_key` string, required — API key to use. Find it at https://app.incident.io/settings/api-keys
- `sourceType` 'incident-io', required
- SourceInsightly
- `start_date` string, date-time, nullable, required — The date from which you'd like to replicate data for Insightly in the format YYYY-MM-DDT00:00:00Z. All data generated after this date will be replicated. Note that it will be used only for incremental streams.
- `token` string, nullable, required — Your Insightly API token.
- `sourceType` 'insightly', required
- SourceInstagram
- `start_date` string, date-time — The date from which you'd like to replicate data for User Insights, in the format YYYY-MM-DDT00:00:00Z. All data generated after this date will be replicated. If left blank, the start date will be set to 2 years before the present date.
- `access_token` string, required — The value of the access token generated with instagram_basic, instagram_manage_insights, pages_show_list, pages_read_engagement, Instagram Public Content Access permissions. See the docs for more information
- `sourceType` 'instagram', required
- SourceInstatus
- `api_key` string, required — Instatus REST API key
- `sourceType` 'instatus', required
- SourceIntercom
- `start_date` string, date-time, required — UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.
- `access_token` string, required — Access token for making authenticated requests. See the Intercom docs for more information.
- `client_id` string — Client Id for your Intercom application.
- `client_secret` string — Client Secret for your Intercom application.
- `activity_logs_time_step` integer — Set lower value in case of failing long running sync of Activity Logs stream.
- `sourceType` 'intercom', required
- SourceIp2whois
- `api_key` string — Your API Key. See here.
- `domain` string — Domain name. See here.
- `sourceType` 'ip2whois', required
- SourceIterable
- `api_key` string, required — Iterable API Key. See the docs for more information on how to obtain this key.
- `start_date` string, date-time, required — The date from which you'd like to replicate data for Iterable, in the format YYYY-MM-DDT00:00:00Z. All data generated after this date will be replicated.
- `sourceType` 'iterable', required
- SourceJira
- `api_token` string, required — Jira API Token. See the docs for more information on how to generate this key. API Token is used for Authorization to your account by BasicAuth.
- `domain` string, required — The Domain for your Jira account, e.g. airbyteio.atlassian.net, airbyteio.jira.com, jira.your-domain.com
- `email` string, required — The user email for your Jira account which you used to generate the API token. This field is used for Authorization to your account by BasicAuth.
- `projects` string[] — List of Jira project keys to replicate data for, or leave it empty if you want to replicate data for all projects.
- `start_date` string, date-time — The date from which you want to replicate data from Jira, use the format YYYY-MM-DDT00:00:00Z. Note that this field only applies to certain streams, and only data generated on or after the start date will be replicated. Or leave it empty if you want to replicate all data. For more information, refer to the documentation.
- `lookback_window_minutes` integer — When set to N, the connector will always refresh resources created within the past N minutes. By default, updated objects that are not newly created are not incrementally synced.
- `enable_experimental_streams` boolean — Allow the use of experimental streams which rely on undocumented Jira API endpoints. See https://docs.airbyte.com/integrations/sources/jira#experimental-tables for more info.
- `sourceType` 'jira', required
- SourceJotform
- `api_key` string, required
- `end_date` string, date-time, required
- `start_date` string, date-time, required
- `api_endpoint` union, required
- object
- `url_prefix` 'Standard' | 'EU' | 'HIPAA', required — You can access our API through the following URLs - Standard API Usage (Use the default API URL - https://api.jotform.com), For EU (Use the EU API URL - https://eu-api.jotform.com), For HIPAA (Use the HIPAA API URL - https://hipaa-api.jotform.com)
- `api_endpoint` 'basic'
- object
- `api_endpoint` 'enterprise'
- `enterprise_url` string, required — Upgrade to Enterprise to make your API url your-domain.com/API or subdomain.jotform.com/API instead of api.jotform.com
- `sourceType` 'jotform', required
- SourceK6Cloud
- `api_token` string, required — Your API Token. See here. The key is case sensitive.
- `sourceType` 'k6-cloud', required
- SourceKissmetrics
- `username` string, required
- `password` string
- `sourceType` 'kissmetrics', required
- SourceKlarna
- `region` 'eu' | 'na' | 'oc', required — Base url region (For playground eu https://docs.klarna.com/klarna-payments/api/payments-api/#tag/API-URLs). Supported 'eu', 'na', 'oc'
- `playground` boolean, required — Propertie defining if connector is used against playground or production environment
- `username` string, required — Consists of your Merchant ID (eid) - a unique number that identifies your e-store, combined with a random string (https://developers.klarna.com/api/#authentication)
- `password` string, required — A string which is associated with your Merchant ID and is used to authorize use of Klarna's APIs (https://developers.klarna.com/api/#authentication)
- `sourceType` 'klarna', required
- SourceKlausApi
- `api_key` string, required — API access key used to retrieve data from the KLAUS API.
- `account` integer, required — getting data by account
- `workspace` integer, required — getting data by workspace
- `start_date` string, date-time — Start getting data from that date.
- `sourceType` 'klaus-api', required
- SourceKlaviyo
- `api_key` string, required — Klaviyo API Key. See our docs if you need help finding this key.
- `start_date` string, date-time — UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated. This field is optional - if not provided, all data will be replicated.
- `disable_fetching_predictive_analytics` boolean — Certain streams like the profiles stream can retrieve predictive analytics data from Klaviyo's API. However, at high volume, this can lead to service availability issues on the API which can be improved by not fetching this field. WARNING: Enabling this setting will stop the "predictive_analytics" column from being populated in your downstream destination.
- `sourceType` 'klaviyo', required
- SourceKyve
- `pool_ids` string, required — The IDs of the KYVE storage pool you want to archive. (Comma separated)
- `start_ids` string, required — The start-id defines, from which bundle id the pipeline should start to extract the data. (Comma separated)
- `url_base` string, required — URL to the KYVE Chain API.
- `sourceType` 'kyve', required
- SourceLaunchdarkly
- `access_token` string, required — Your Access token. See here.
- `sourceType` 'launchdarkly', required
- SourceLeadfeeder
- `api_token` string, required
- `start_date` string, date-time, required
- `sourceType` 'leadfeeder', required
- SourceLemlist
- `api_key` string, required — Lemlist API key,
- `sourceType` 'lemlist', required
- SourceLeverHiring
- `credentials` union — Choose how to authenticate to Lever Hiring.
- object
- `auth_type` 'Client'
- `client_id` string — The Client ID of your Lever Hiring developer application.
- `client_secret` string — The Client Secret of your Lever Hiring developer application.
- `refresh_token` string, required — The token for obtaining new access token.
- object
- `auth_type` 'Api Key'
- `api_key` string, required — The Api Key of your Lever Hiring account.
- `start_date` string, required — UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated. Note that it will be used only in the following incremental streams: comments, commits, and issues.
- `environment` 'Production' | 'Sandbox' — The environment in which you'd like to replicate data for Lever. This is used to determine which Lever API endpoint to use.
- `sourceType` 'lever-hiring', required
- SourceLinkedinAds
- `credentials` union
- object
- `auth_method` 'oAuth2.0'
- `client_id` string, required — The client ID of your developer application. Refer to our documentation for more information.
- `client_secret` string, required — The client secret of your developer application. Refer to our documentation for more information.
- `refresh_token` string, required — The key to refresh the expired access token. Refer to our documentation for more information.
- object
- `auth_method` 'access_token'
- `access_token` string, required — The access token generated for your developer application. Refer to our documentation for more information.
- `start_date` string, date, required — UTC date in the format YYYY-MM-DD. Any data before this date will not be replicated.
- `lookback_window` integer — How far into the past to look for records. (in days)
- `account_ids` integer[] — Specify the account IDs to pull data from, separated by a space. Leave this field empty if you want to pull the data from all accounts accessible by the authenticated user. See the LinkedIn docs to locate these IDs.
- `ad_analytics_reports` object[]
- `name` string, required — The name for the custom report.
- `pivot_by` 'COMPANY' | 'ACCOUNT' | 'SHARE' | 'CAMPAIGN' | 'CREATIVE' | 'CAMPAIGN_GROUP' | 'CONVERSION' | 'CONVERSATION_NODE' | 'CONVERSATION_NODE_OPTION_INDEX' | 'SERVING_LOCATION' | 'CARD_INDEX' | 'MEMBER_COMPANY_SIZE' | 'MEMBER_INDUSTRY' | 'MEMBER_SENIORITY' | 'MEMBER_JOB_TITLE' | 'MEMBER_JOB_FUNCTION' | 'MEMBER_COUNTRY_V2' | 'MEMBER_REGION_V2' | 'MEMBER_COMPANY' | 'PLACEMENT_NAME' | 'IMPRESSION_DEVICE_TYPE', required — Choose a category to pivot your analytics report around. This selection will organize your data based on the chosen attribute, allowing you to analyze trends and performance from different perspectives.
- `time_granularity` 'ALL' | 'DAILY' | 'MONTHLY' | 'YEARLY', required — Choose how to group the data in your report by time. The options are:
- 'ALL': A single result summarizing the entire time range.
- 'DAILY': Group results by each day.
- 'MONTHLY': Group results by each month.
- 'YEARLY': Group results by each year.
Selecting a time grouping helps you analyze trends and patterns over different time periods.
- `sourceType` 'linkedin-ads', required
- SourceLinkedinPages
- `credentials` union
- object
- `client_id` string, required — The client ID of the LinkedIn developer application.
- `auth_method` 'oAuth2.0'
- `client_secret` string, required — The client secret of the LinkedIn developer application.
- `refresh_token` string, required — The token value generated using the LinkedIn Developers OAuth Token Tools. See the docs to obtain yours.
- object
- `auth_method` 'access_token'
- `access_token` string, required — The token value generated using the LinkedIn Developers OAuth Token Tools. See the docs to obtain yours.
- `org_id` string, required — Specify the Organization ID
- `start_date` string, date-time — Start date for getting metrics per time period. Must be atmost 12 months before the request date (UTC) and atleast 2 days prior to the request date (UTC). See https://bit.ly/linkedin-pages-date-rules {{ "\n" }} {{ response.errorDetails }}
- `time_granularity_type` 'DAY' | 'MONTH' — Granularity of the statistics for metrics per time period. Must be either "DAY" or "MONTH"
- `sourceType` 'linkedin-pages', required
- SourceLinnworks
- `application_id` string, required — Linnworks Application ID
- `application_secret` string, required — Linnworks Application Secret
- `token` string, required
- `start_date` string, date-time, required — UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.
- `sourceType` 'linnworks', required
- SourceLob
- `api_key` string, required — API key to use for authentication. You can find your account's API keys in your Dashboard Settings at https://dashboard.lob.com/settings/api-keys.
- `start_date` string, date-time, required
- `limit` string — Max records per page limit
- `sourceType` 'lob', required
- SourceLokalise
- `api_key` string, required — Lokalise API Key with read-access. Available at Profile settings > API tokens. See here.
- `project_id` string, required — Lokalise project ID. Available at Project Settings > General.
- `sourceType` 'lokalise', required
- SourceLooker
- `client_id` string, required — The Client ID is first part of an API3 key that is specific to each Looker user. See the docs for more information on how to generate this key.
- `client_secret` string, required — The Client Secret is second part of an API3 key.
- `domain` string, required — Domain for your Looker account, e.g. airbyte.cloud.looker.com,looker.[clientname].com,IP address
- `run_look_ids` string[] — The IDs of any Looks to run
- `sourceType` 'looker', required
- SourceLuma
- `api_key` string, required — Get your API key on lu.ma Calendars dashboard → Settings.
- `sourceType` 'luma', required
- SourceMailchimp
- `credentials` union
- object
- `auth_type` 'oauth2.0', required
- `client_id` string — The Client ID of your OAuth application.
- `client_secret` string — The Client Secret of your OAuth application.
- `access_token` string, required — An access token generated using the above client ID and secret.
- object
- `auth_type` 'apikey', required
- `apikey` string, required — Mailchimp API Key. See the docs for information on how to generate this key.
- `start_date` string, date-time — The date from which you want to start syncing data for Incremental streams. Only records that have been created or modified since this date will be synced. If left blank, all data will by synced.
- `sourceType` 'mailchimp', required
- SourceMailerlite
- `api_token` string, required — Your API Token. See here.
- `sourceType` 'mailerlite', required
- SourceMailgun
- `private_key` string, required — Primary account API key to access your Mailgun data.
- `domain_region` 'US' | 'EU' — Domain region code. 'EU' or 'US' are possible values. The default is 'US'.
- `start_date` string, date-time — UTC date and time in the format 2020-10-01 00:00:00. Any data before this date will not be replicated. If omitted, defaults to 3 days ago.
- `sourceType` 'mailgun', required
- SourceMailjetMail
- `api_key` string, required — Your API Key. See here.
- `api_key_secret` string, required — Your API Secret Key. See here.
- `sourceType` 'mailjet-mail', required
- SourceMailjetSms
- `end_date` integer — Retrieve SMS messages created before the specified timestamp. Required format - Unix timestamp.
- `start_date` integer — Retrieve SMS messages created after the specified timestamp. Required format - Unix timestamp.
- `token` string, required — Your access token. See here.
- `sourceType` 'mailjet-sms', required
- SourceMarketo
- `domain_url` string, required — Your Marketo Base URL. See the docs for info on how to obtain this.
- `client_id` string, required — The Client ID of your Marketo developer application. See the docs for info on how to obtain this.
- `client_secret` string, required — The Client Secret of your Marketo developer application. See the docs for info on how to obtain this.
- `start_date` string, date-time, required — UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.
- `sourceType` 'marketo', required
- SourceMetabase
- `instance_api_url` string, required — URL to your metabase instance API
- `username` string, required
- `password` string
- `session_token` string — To generate your session token, you need to run the following command: ``` curl -X POST \ -H "Content-Type: application/json" \ -d '{"username": "person@metabase.com", "password": "fakepassword"}' \ http://localhost:3000/api/session ``` Then copy the value of the `id` field returned by a successful call to that API. Note that by default, sessions are good for 14 days and needs to be regenerated.
- `sourceType` 'metabase', required
- SourceMicrosoftDataverse
- `url` string, required — URL to Microsoft Dataverse API
- `tenant_id` string, required — Tenant Id of your Microsoft Dataverse Instance
- `client_id` string, required — App Registration Client Id
- `client_secret_value` string, required — App Registration Client Secret
- `odata_maxpagesize` integer — Max number of results per page. Default=5000
- `sourceType` 'microsoft-dataverse', required
- SourceMicrosoftOnedrive — SourceMicrosoftOneDriveSpec class for Microsoft OneDrive Source Specification. This class combines the authentication details with additional configuration for the OneDrive API.
- `start_date` string, date-time — UTC date and time in the format 2017-01-25T00:00:00.000000Z. Any file modified before this date will not be replicated.
- `streams` object[], required — Each instance of this configuration defines a stream. Use this to define which files belong in the stream, their format, and how they should be parsed and validated. When sending data to warehouse destination such as Snowflake or BigQuery, each stream is a separate table.
- `name` string, required — The name of the stream.
- `globs` string[] — The pattern used to specify which files should be selected from the file system. For more information on glob pattern matching look here.
- `validation_policy` 'Emit Record' | 'Skip Record' | 'Wait for Discover' — The name of the validation policy that dictates sync behavior when a record does not adhere to the stream schema.
- `input_schema` string — The schema that will be used to validate records extracted from the file. This will override the stream schema that is auto-detected from incoming files.
- `days_to_sync_if_history_is_full` integer — When the state history of the file store is full, syncs will only read files that were last modified in the provided day range.
- `format` union, required — The configuration options that are used to alter how to read incoming files that deviate from the standard formatting.
- object
- `filetype` 'avro', required
- `double_as_string` boolean — Whether to convert double fields to strings. This is recommended if you have decimal numbers with a high degree of precision because there can be a loss precision when handling floating point numbers.
- object
- `filetype` 'csv', required
- `delimiter` string — The character delimiting individual cells in the CSV data. This may only be a 1-character string. For tab-delimited data enter '\t'.
- `quote_char` string — The character used for quoting CSV values. To disallow quoting, make this field blank.
- `escape_char` string — The character used for escaping special characters. To disallow escaping, leave this field blank.
- `encoding` string — The character encoding of the CSV data. Leave blank to default to UTF8. See list of python encodings for allowable options.
- `double_quote` boolean — Whether two quotes in a quoted CSV value denote a single quote in the data.
- `null_values` string[] — A set of case-sensitive strings that should be interpreted as null values. For example, if the value 'NA' should be interpreted as null, enter 'NA' in this field.
- `strings_can_be_null` boolean — Whether strings can be interpreted as null values. If true, strings that match the null_values set will be interpreted as null. If false, strings that match the null_values set will be interpreted as the string itself.
- `skip_rows_before_header` integer — The number of rows to skip before the header row. For example, if the header row is on the 3rd row, enter 2 in this field.
- `skip_rows_after_header` integer — The number of rows to skip after the header row.
- `header_definition` union — How headers will be defined. `User Provided` assumes the CSV does not have a header row and uses the headers provided and `Autogenerated` assumes the CSV does not have a header row and the CDK will generate headers using for `f{i}` where `i` is the index starting from 0. Else, the default behavior is to use the header from the CSV file. If a user wants to autogenerate or provide column names for a CSV having headers, they can skip rows.
- …
- `true_values` string[] — A set of case-sensitive strings that should be interpreted as true values.
- `false_values` string[] — A set of case-sensitive strings that should be interpreted as false values.
- `ignore_errors_on_fields_mismatch` boolean — Whether to ignore errors that occur when the number of fields in the CSV does not match the number of columns in the schema.
- object
- `filetype` 'jsonl', required
- object
- `filetype` 'parquet', required
- `decimal_as_float` boolean — Whether to convert decimal fields to floats. There is a loss of precision when converting decimals to floats, so this is not recommended.
- object — Extract text from document formats (.pdf, .docx, .md, .pptx) and emit as one record per file.
- `filetype` 'unstructured', required
- `skip_unprocessable_files` boolean — If true, skip files that cannot be parsed and pass the error message along as the _ab_source_file_parse_error field. If false, fail the sync.
- `strategy` 'auto' | 'fast' | 'ocr_only' | 'hi_res' — The strategy used to parse documents. `fast` extracts text directly from the document which doesn't work for all files. `ocr_only` is more reliable, but slower. `hi_res` is the most reliable, but requires an API key and a hosted instance of unstructured and can't be used with local mode. See the unstructured.io documentation for more details: https://unstructured-io.github.io/unstructured/core/partition.html#partition-pdf
- `processing` object — Process files locally, supporting `fast` and `ocr` modes. This is the default option.
- …
- `schemaless` boolean — When enabled, syncs will not validate or structure records against the stream's schema.
- `credentials` union, required — Credentials for connecting to the One Drive API
- object — OAuthCredentials class to hold authentication details for Microsoft OAuth authentication. This class uses pydantic for data validation and settings management.
- `auth_type` 'Client'
- `tenant_id` string, required — Tenant ID of the Microsoft OneDrive user
- `client_id` string, required — Client ID of your Microsoft developer application
- `client_secret` string, required — Client Secret of your Microsoft developer application
- `refresh_token` string, required — Refresh Token of your Microsoft developer application
- object — ServiceCredentials class for service key authentication. This class is structured similarly to OAuthCredentials but for a different authentication method.
- `auth_type` 'Service'
- `tenant_id` string, required — Tenant ID of the Microsoft OneDrive user
- `user_principal_name` string, required — Special characters such as a period, comma, space, and the at sign (@) are converted to underscores (_). More details: https://learn.microsoft.com/en-us/sharepoint/list-onedrive-urls
- `client_id` string, required — Client ID of your Microsoft developer application
- `client_secret` string, required — Client Secret of your Microsoft developer application
- `drive_name` string — Name of the Microsoft OneDrive drive where the file(s) exist.
- `search_scope` 'ACCESSIBLE_DRIVES' | 'SHARED_ITEMS' | 'ALL' — Specifies the location(s) to search for files. Valid options are 'ACCESSIBLE_DRIVES' to search in the selected OneDrive drive, 'SHARED_ITEMS' for shared items the user has access to, and 'ALL' to search both.
- `folder_path` string — Path to a specific folder within the drives to search for files. Leave empty to search all folders of the drives. This does not apply to shared items.
- `sourceType` 'microsoft-onedrive', required
- SourceMicrosoftSharepoint — SourceMicrosoftSharePointSpec class for Microsoft SharePoint Source Specification. This class combines the authentication details with additional configuration for the SharePoint API.
- `start_date` string, date-time — UTC date and time in the format 2017-01-25T00:00:00.000000Z. Any file modified before this date will not be replicated.
- `streams` object[], required — Each instance of this configuration defines a stream. Use this to define which files belong in the stream, their format, and how they should be parsed and validated. When sending data to warehouse destination such as Snowflake or BigQuery, each stream is a separate table.
- `name` string, required — The name of the stream.
- `globs` string[] — The pattern used to specify which files should be selected from the file system. For more information on glob pattern matching look here.
- `validation_policy` 'Emit Record' | 'Skip Record' | 'Wait for Discover' — The name of the validation policy that dictates sync behavior when a record does not adhere to the stream schema.
- `input_schema` string — The schema that will be used to validate records extracted from the file. This will override the stream schema that is auto-detected from incoming files.
- `days_to_sync_if_history_is_full` integer — When the state history of the file store is full, syncs will only read files that were last modified in the provided day range.
- `format` union, required — The configuration options that are used to alter how to read incoming files that deviate from the standard formatting.
- object
- `filetype` 'avro', required
- `double_as_string` boolean — Whether to convert double fields to strings. This is recommended if you have decimal numbers with a high degree of precision because there can be a loss precision when handling floating point numbers.
- object
- `filetype` 'csv', required
- `delimiter` string — The character delimiting individual cells in the CSV data. This may only be a 1-character string. For tab-delimited data enter '\t'.
- `quote_char` string — The character used for quoting CSV values. To disallow quoting, make this field blank.
- `escape_char` string — The character used for escaping special characters. To disallow escaping, leave this field blank.
- `encoding` string — The character encoding of the CSV data. Leave blank to default to UTF8. See list of python encodings for allowable options.
- `double_quote` boolean — Whether two quotes in a quoted CSV value denote a single quote in the data.
- `null_values` string[] — A set of case-sensitive strings that should be interpreted as null values. For example, if the value 'NA' should be interpreted as null, enter 'NA' in this field.
- `strings_can_be_null` boolean — Whether strings can be interpreted as null values. If true, strings that match the null_values set will be interpreted as null. If false, strings that match the null_values set will be interpreted as the string itself.
- `skip_rows_before_header` integer — The number of rows to skip before the header row. For example, if the header row is on the 3rd row, enter 2 in this field.
- `skip_rows_after_header` integer — The number of rows to skip after the header row.
- `header_definition` union — How headers will be defined. `User Provided` assumes the CSV does not have a header row and uses the headers provided and `Autogenerated` assumes the CSV does not have a header row and the CDK will generate headers using for `f{i}` where `i` is the index starting from 0. Else, the default behavior is to use the header from the CSV file. If a user wants to autogenerate or provide column names for a CSV having headers, they can skip rows.
- …
- `true_values` string[] — A set of case-sensitive strings that should be interpreted as true values.
- `false_values` string[] — A set of case-sensitive strings that should be interpreted as false values.
- `ignore_errors_on_fields_mismatch` boolean — Whether to ignore errors that occur when the number of fields in the CSV does not match the number of columns in the schema.
- object
- `filetype` 'jsonl', required
- object
- `filetype` 'parquet', required
- `decimal_as_float` boolean — Whether to convert decimal fields to floats. There is a loss of precision when converting decimals to floats, so this is not recommended.
- object — Extract text from document formats (.pdf, .docx, .md, .pptx) and emit as one record per file.
- `filetype` 'unstructured', required
- `skip_unprocessable_files` boolean — If true, skip files that cannot be parsed and pass the error message along as the _ab_source_file_parse_error field. If false, fail the sync.
- `strategy` 'auto' | 'fast' | 'ocr_only' | 'hi_res' — The strategy used to parse documents. `fast` extracts text directly from the document which doesn't work for all files. `ocr_only` is more reliable, but slower. `hi_res` is the most reliable, but requires an API key and a hosted instance of unstructured and can't be used with local mode. See the unstructured.io documentation for more details: https://unstructured-io.github.io/unstructured/core/partition.html#partition-pdf
- `processing` object — Process files locally, supporting `fast` and `ocr` modes. This is the default option.
- …
- object
- `filetype` 'excel', required
- `schemaless` boolean — When enabled, syncs will not validate or structure records against the stream's schema.
- `recent_n_files_to_read_for_schema_discovery` integer — The number of resent files which will be used to discover the schema for this stream.
- `credentials` union, required — Credentials for connecting to the One Drive API
- object — OAuthCredentials class to hold authentication details for Microsoft OAuth authentication. This class uses pydantic for data validation and settings management.
- `auth_type` 'Client'
- `tenant_id` string, required — Tenant ID of the Microsoft SharePoint user
- `client_id` string, required — Client ID of your Microsoft developer application
- `client_secret` string, required — Client Secret of your Microsoft developer application
- `refresh_token` string — Refresh Token of your Microsoft developer application
- object — ServiceCredentials class for service key authentication. This class is structured similarly to OAuthCredentials but for a different authentication method.
- `auth_type` 'Service'
- `tenant_id` string, required — Tenant ID of the Microsoft SharePoint user
- `user_principal_name` string, required — Special characters such as a period, comma, space, and the at sign (@) are converted to underscores (_). More details: https://learn.microsoft.com/en-us/sharepoint/list-onedrive-urls
- `client_id` string, required — Client ID of your Microsoft developer application
- `client_secret` string, required — Client Secret of your Microsoft developer application
- `search_scope` 'ACCESSIBLE_DRIVES' | 'SHARED_ITEMS' | 'ALL' — Specifies the location(s) to search for files. Valid options are 'ACCESSIBLE_DRIVES' for all SharePoint drives the user can access, 'SHARED_ITEMS' for shared items the user has access to, and 'ALL' to search both.
- `folder_path` string — Path to a specific folder within the drives to search for files. Leave empty to search all folders of the drives. This does not apply to shared items.
- `sourceType` 'microsoft-sharepoint', required
- SourceMicrosoftTeams
- `period` string, required — Specifies the length of time over which the Team Device Report stream is aggregated. The supported values are: D7, D30, D90, and D180.
- `credentials` union — Choose how to authenticate to Microsoft
- object
- `auth_type` 'Client'
- `tenant_id` string, required — A globally unique identifier (GUID) that is different than your organization name or domain. Follow these steps to obtain: open one of the Teams where you belong inside the Teams Application -> Click on the … next to the Team title -> Click on Get link to team -> Copy the link to the team and grab the tenant ID form the URL
- `client_id` string, required — The Client ID of your Microsoft Teams developer application.
- `client_secret` string, required — The Client Secret of your Microsoft Teams developer application.
- `refresh_token` string, required — A Refresh Token to renew the expired Access Token.
- object
- `auth_type` 'Token'
- `tenant_id` string, required — A globally unique identifier (GUID) that is different than your organization name or domain. Follow these steps to obtain: open one of the Teams where you belong inside the Teams Application -> Click on the … next to the Team title -> Click on Get link to team -> Copy the link to the team and grab the tenant ID form the URL
- `client_id` string, required — The Client ID of your Microsoft Teams developer application.
- `client_secret` string, required — The Client Secret of your Microsoft Teams developer application.
- `sourceType` 'microsoft-teams', required
- SourceMissive
- `api_key` string, required
- `limit` string — Max records per page limit
- `start_date` string, date-time, required
- `kind` 'group' | 'organization' — Kind parameter for `contact_groups` stream
- `sourceType` 'missive', required
- SourceMixmax
- `api_key` string, required
- `start_date` string, date-time, required
- `sourceType` 'mixmax', required
- SourceMixpanel
- `credentials` union, required — Choose how to authenticate to Mixpanel
- object
- `option_title` 'Service Account'
- `username` string, required — Mixpanel Service Account Username. See the docs for more information on how to obtain this.
- `secret` string, required — Mixpanel Service Account Secret. See the docs for more information on how to obtain this.
- `project_id` integer, required — Your project ID number. See the docs for more information on how to obtain this.
- object
- `option_title` 'Project Secret'
- `api_secret` string, required — Mixpanel project secret. See the docs for more information on how to obtain this.
- `attribution_window` integer — A period of time for attributing results to ads and the lookback period after those actions occur during which ad results are counted. Default attribution window is 5 days. (This value should be non-negative integer)
- `project_timezone` string — Time zone in which integer date times are stored. The project timezone may be found in the project settings in the Mixpanel console.
- `select_properties_by_default` boolean — Setting this config parameter to TRUE ensures that new properties on events and engage records are captured. Otherwise new properties will be ignored.
- `start_date` string, date-time — The date in the format YYYY-MM-DD. Any data before this date will not be replicated. If this option is not set, the connector will replicate data from up to one year ago by default.
- `end_date` string, date-time — The date in the format YYYY-MM-DD. Any data after this date will not be replicated. Left empty to always sync to most recent date
- `region` 'US' | 'EU' — The region of mixpanel domain instance either US or EU.
- `date_window_size` integer — Defines window size in days, that used to slice through data. You can reduce it, if amount of data in each window is too big for your environment. (This value should be positive integer)
- `page_size` integer — The number of records to fetch per request for the engage stream. Default is 1000. If you are experiencing long sync times with this stream, try increasing this value.
- `sourceType` 'mixpanel', required
- SourceMonday
- `credentials` union
- object
- `subdomain` string — Slug/subdomain of the account, or the first part of the URL that comes before .monday.com
- `auth_type` 'oauth2.0', required
- `client_id` string, required — The Client ID of your OAuth application.
- `client_secret` string, required — The Client Secret of your OAuth application.
- `access_token` string, required — Access Token for making authenticated requests.
- object
- `auth_type` 'api_token', required
- `api_token` string, required — API Token for making authenticated requests.
- `sourceType` 'monday', required
- SourceMongodbV2
- `database_config` union, required — Configures the MongoDB cluster type.
- object — MongoDB Atlas-hosted cluster configured as a replica set
- `cluster_type` 'ATLAS_REPLICA_SET', required
- `connection_string` string, required — The connection string of the cluster that you want to replicate.
- `database` string, required — The name of the MongoDB database that contains the collection(s) to replicate.
- `username` string, required — The username which is used to access the database.
- `password` string, required — The password associated with this username.
- `auth_source` string, required — The authentication source where the user information is stored. See https://www.mongodb.com/docs/manual/reference/connection-string/#mongodb-urioption-urioption.authSource for more details.
- `schema_enforced` boolean — When enabled, syncs will validate and structure records against the stream's schema.
- object — MongoDB self-hosted cluster configured as a replica set
- `cluster_type` 'SELF_MANAGED_REPLICA_SET', required
- `connection_string` string, required — The connection string of the cluster that you want to replicate. https://www.mongodb.com/docs/manual/reference/connection-string/#find-your-self-hosted-deployment-s-connection-string for more information.
- `database` string, required — The name of the MongoDB database that contains the collection(s) to replicate.
- `username` string — The username which is used to access the database.
- `password` string — The password associated with this username.
- `auth_source` string — The authentication source where the user information is stored.
- `schema_enforced` boolean — When enabled, syncs will validate and structure records against the stream's schema.
- `initial_waiting_seconds` integer — The amount of time the connector will wait when it launches to determine if there is new data to sync or not. Defaults to 300 seconds. Valid range: 120 seconds to 1200 seconds.
- `queue_size` integer — The size of the internal queue. This may interfere with memory consumption and efficiency of the connector, please be careful.
- `discover_sample_size` integer — The maximum number of documents to sample when attempting to discover the unique fields for a collection.
- `invalid_cdc_cursor_position_behavior` 'Fail sync' | 'Re-sync data' — Determines whether Airbyte should fail or re-sync data in case of an stale/invalid cursor value into the WAL. If 'Fail sync' is chosen, a user will have to manually reset the connection before being able to continue syncing data. If 'Re-sync data' is chosen, Airbyte will automatically trigger a refresh but could lead to higher cloud costs and data loss.
- `update_capture_mode` 'Lookup' | 'Post Image' — Determines how Airbyte looks up the value of an updated document. If 'Lookup' is chosen, the current value of the document will be read. If 'Post Image' is chosen, then the version of the document immediately after an update will be read. WARNING : Severe data loss will occur if this option is chosen and the appropriate settings are not set on your Mongo instance : https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre-and-post-images.
- `initial_load_timeout_hours` integer — The amount of time an initial load is allowed to continue for before catching up on CDC logs.
- `sourceType` 'mongodb-v2', required
- SourceMssql
- `host` string, required — The hostname of the database.
- `port` integer, required — The port of the database.
- `database` string, required — The name of the database.
- `schemas` string[] — The list of schemas to sync from. Defaults to user. Case sensitive.
- `username` string, required — The username which is used to access the database.
- `password` string, required — The password associated with the username.
- `jdbc_url_params` string — Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (example: key1=value1&key2=value2&key3=value3).
- `ssl_method` union — The encryption method which is used when communicating with the database.
- object — Data transfer will not be encrypted.
- `ssl_method` 'unencrypted', required
- object — Use the certificate provided by the server without verification. (For testing purposes only!)
- `ssl_method` 'encrypted_trust_server_certificate', required
- object — Verify and use the certificate provided by the server.
- `ssl_method` 'encrypted_verify_certificate', required
- `hostNameInCertificate` string — Specifies the host name of the server. The value of this property must match the subject property of the certificate.
- `certificate` string — certificate of the server, or of the CA that signed the server certificate
- `replication_method` union — Configures how data is extracted from the database.
- object — Recommended - Incrementally reads new inserts, updates, and deletes using the SQL Server's change data capture feature. This must be enabled on your database.
- `method` 'CDC', required
- `initial_waiting_seconds` integer — The amount of time the connector will wait when it launches to determine if there is new data to sync or not. Defaults to 300 seconds. Valid range: 120 seconds to 3600 seconds. Read about initial waiting time.
- `invalid_cdc_cursor_position_behavior` 'Fail sync' | 'Re-sync data' — Determines whether Airbyte should fail or re-sync data in case of an stale/invalid cursor value into the WAL. If 'Fail sync' is chosen, a user will have to manually reset the connection before being able to continue syncing data. If 'Re-sync data' is chosen, Airbyte will automatically trigger a refresh but could lead to higher cloud costs and data loss.
- `queue_size` integer — The size of the internal queue. This may interfere with memory consumption and efficiency of the connector, please be careful.
- `initial_load_timeout_hours` integer — The amount of time an initial load is allowed to continue for before catching up on CDC logs.
- object — Incrementally detects new inserts and updates using the cursor column chosen when configuring a connection (e.g. created_at, updated_at).
- `method` 'STANDARD', required
- `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use.
- object
- `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database
- object
- `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key
- `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel.
- `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections.
- `tunnel_user` string, required — OS-level username for logging into the jump server host.
- `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa )
- object
- `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication
- `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel.
- `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections.
- `tunnel_user` string, required — OS-level username for logging into the jump server host
- `tunnel_user_password` string, required — OS-level password for logging into the jump server host
- `sourceType` 'mssql', required
- SourceMux
- `username` string, required
- `password` string
- `start_date` string, date-time, required
- `playback_id` string — The playback id for your video asset shown in website details
- `sourceType` 'mux', required
- SourceMyHours
- `email` string, required — Your My Hours username
- `password` string, required — The password associated to the username
- `start_date` string, required — Start date for collecting time logs
- `logs_batch_size` integer — Pagination size used for retrieving logs in days
- `sourceType` 'my-hours', required
- SourceMysql
- `host` string, required — The host name of the database.
- `port` integer, required — The port to connect to.
- `database` string, required — The database name.
- `username` string, required — The username which is used to access the database.
- `password` string — The password associated with the username.
- `jdbc_url_params` string — Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (example: key1=value1&key2=value2&key3=value3). For more information read about JDBC URL parameters.
- `ssl` boolean — Encrypt data using SSL.
- `ssl_mode` union — SSL connection modes. Read more in the docs.
- object — Automatically attempt SSL connection. If the MySQL server does not support SSL, continue with a regular connection.
- `mode` 'preferred', required
- object — Always connect with SSL. If the MySQL server doesn’t support SSL, the connection will not be established. Certificate Authority (CA) and Hostname are not verified.
- `mode` 'required', required
- object — Always connect with SSL. Verifies CA, but allows connection even if Hostname does not match.
- `mode` 'verify_ca', required
- `ca_certificate` string, required — CA certificate
- `client_certificate` string — Client certificate (this is not a required field, but if you want to use it, you will need to add the Client key as well)
- `client_key` string — Client key (this is not a required field, but if you want to use it, you will need to add the Client certificate as well)
- `client_key_password` string — Password for keystorage. This field is optional. If you do not add it - the password will be generated automatically.
- object — Always connect with SSL. Verify both CA and Hostname.
- `mode` 'verify_identity', required
- `ca_certificate` string, required — CA certificate
- `client_certificate` string — Client certificate (this is not a required field, but if you want to use it, you will need to add the Client key as well)
- `client_key` string — Client key (this is not a required field, but if you want to use it, you will need to add the Client certificate as well)
- `client_key_password` string — Password for keystorage. This field is optional. If you do not add it - the password will be generated automatically.
- `replication_method` union, required — Configures how data is extracted from the database.
- object — Recommended - Incrementally reads new inserts, updates, and deletes using the MySQL binary log. This must be enabled on your database.
- `method` 'CDC', required
- `initial_waiting_seconds` integer — The amount of time the connector will wait when it launches to determine if there is new data to sync or not. Defaults to 300 seconds. Valid range: 120 seconds to 1200 seconds. Read about initial waiting time.
- `server_time_zone` string — Enter the configured MySQL server timezone. This should only be done if the configured timezone in your MySQL instance does not conform to IANNA standard.
- `invalid_cdc_cursor_position_behavior` 'Fail sync' | 'Re-sync data' — Determines whether Airbyte should fail or re-sync data in case of an stale/invalid cursor value into the WAL. If 'Fail sync' is chosen, a user will have to manually reset the connection before being able to continue syncing data. If 'Re-sync data' is chosen, Airbyte will automatically trigger a refresh but could lead to higher cloud costs and data loss.
- `initial_load_timeout_hours` integer — The amount of time an initial load is allowed to continue for before catching up on CDC logs.
- object — Incrementally detects new inserts and updates using the cursor column chosen when configuring a connection (e.g. created_at, updated_at).
- `method` 'STANDARD', required
- `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use.
- object
- `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database
- object
- `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key
- `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel.
- `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections.
- `tunnel_user` string, required — OS-level username for logging into the jump server host.
- `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa )
- object
- `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication
- `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel.
- `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections.
- `tunnel_user` string, required — OS-level username for logging into the jump server host
- `tunnel_user_password` string, required — OS-level password for logging into the jump server host
- `sourceType` 'mysql', required
- SourceN8n
- `host` string, required — Hostname of the n8n instance
- `api_key` string, required — Your API KEY. See here
- `sourceType` 'n8n', required
- SourceNasa
- `api_key` string, required — API access key used to retrieve data from the NASA APOD API.
- `concept_tags` boolean — Indicates whether concept tags should be returned with the rest of the response. The concept tags are not necessarily included in the explanation, but rather derived from common search tags that are associated with the description text. (Better than just pure text search.) Defaults to False.
- `count` integer — A positive integer, no greater than 100. If this is specified then `count` randomly chosen images will be returned in a JSON array. Cannot be used in conjunction with `date` or `start_date` and `end_date`.
- `end_date` string, date — Indicates that end of a date range. If `start_date` is specified without an `end_date` then `end_date` defaults to the current date.
- `start_date` string, date — Indicates the start of a date range. All images in the range from `start_date` to `end_date` will be returned in a JSON array. Must be after 1995-06-16, the first day an APOD picture was posted. There are no images for tomorrow available through this API.
- `thumbs` boolean — Indicates whether the API should return a thumbnail image URL for video files. If set to True, the API returns URL of video thumbnail. If an APOD is not a video, this parameter is ignored.
- `sourceType` 'nasa', required
- SourceNetsuite
- `realm` string, required — Netsuite realm e.g. 2344535, as for `production` or 2344535_SB1, as for the `sandbox`
- `consumer_key` string, required — Consumer key associated with your integration
- `consumer_secret` string, required — Consumer secret associated with your integration
- `token_key` string, required — Access token key
- `token_secret` string, required — Access token secret
- `object_types` string[] — The API names of the Netsuite objects you want to sync. Setting this speeds up the connection setup process by limiting the number of schemas that need to be retrieved from Netsuite.
- `start_datetime` string, required — Starting point for your data replication, in format of "YYYY-MM-DDTHH:mm:ssZ"
- `window_in_days` integer — The amount of days used to query the data with date chunks. Set smaller value, if you have lots of data.
- `sourceType` 'netsuite', required
- SourceNewsApi
- `api_key` string, required — API Key
- `search_query` string — Search query. See https://newsapi.org/docs/endpoints/everything for information.
- `search_in` string[] — Where to apply search query. Possible values are: title, description, content.
- `sources` string[] — Identifiers (maximum 20) for the news sources or blogs you want headlines from. Use the `/sources` endpoint to locate these programmatically or look at the sources index: https://newsapi.com/sources. Will override both country and category.
- `domains` string[] — A comma-seperated string of domains (eg bbc.co.uk, techcrunch.com, engadget.com) to restrict the search to.
- `exclude_domains` string[] — A comma-seperated string of domains (eg bbc.co.uk, techcrunch.com, engadget.com) to remove from the results.
- `start_date` string — A date and optional time for the oldest article allowed. This should be in ISO 8601 format.
- `end_date` string — A date and optional time for the newest article allowed. This should be in ISO 8601 format.
- `language` 'ar' | 'de' | 'en' | 'es' | 'fr' | 'he' | 'it' | 'nl' | 'no' | 'pt' | 'ru' | 'se' | 'ud' | 'zh' — The 2-letter ISO-639-1 code of the language you want to get headlines for. Possible options: ar de en es fr he it nl no pt ru se ud zh.
- `country` 'ae' | 'ar' | 'at' | 'au' | 'be' | 'bg' | 'br' | 'ca' | 'ch' | 'cn' | 'co' | 'cu' | 'cz' | 'de' | 'eg' | 'fr' | 'gb' | 'gr' | 'hk' | 'hu' | 'id' | 'ie' | 'il' | 'in' | 'it' | 'jp' | 'kr' | 'lt' | 'lv' | 'ma' | 'mx' | 'my' | 'ng' | 'nl' | 'no' | 'nz' | 'ph' | 'pl' | 'pt' | 'ro' | 'rs' | 'ru' | 'sa' | 'se' | 'sg' | 'si' | 'sk' | 'th' | 'tr' | 'tw' | 'ua' | 'us' | 've' | 'za', required — The 2-letter ISO 3166-1 code of the country you want to get headlines for. You can't mix this with the sources parameter.
- `category` 'business' | 'entertainment' | 'general' | 'health' | 'science' | 'sports' | 'technology', required — The category you want to get top headlines for.
- `sort_by` 'relevancy' | 'popularity' | 'publishedAt', required — The order to sort the articles in. Possible options: relevancy, popularity, publishedAt.
- `sourceType` 'news-api', required
- SourceNorthpassLms
- `api_key` string, required
- `sourceType` 'northpass-lms', required
- SourceNotion
- `start_date` string, date-time — UTC date and time in the format YYYY-MM-DDTHH:MM:SS.000Z. During incremental sync, any data generated before this date will not be replicated. If left blank, the start date will be set to 2 years before the present date.
- `credentials` union — Choose either OAuth (recommended for Airbyte Cloud) or Access Token. See our docs for more information.
- object
- `auth_type` 'OAuth2.0', required
- `client_id` string, required — The Client ID of your Notion integration. See our docs for more information.
- `client_secret` string, required — The Client Secret of your Notion integration. See our docs for more information.
- `access_token` string, required — The Access Token received by completing the OAuth flow for your Notion integration. See our docs for more information.
- object
- `auth_type` 'token', required
- `token` string, required — The Access Token for your private Notion integration. See the docs for more information on how to obtain this token.
- `sourceType` 'notion'
- SourceNylas
- `api_key` string, required
- `api_server` 'us' | 'eu', required
- `start_date` string, date-time, required
- `end_date` string, date-time, required
- `sourceType` 'nylas', required
- SourceNytimes
- `api_key` string, required — API Key
- `start_date` string, required — Start date to begin the article retrieval (format YYYY-MM)
- `end_date` string — End date to stop the article retrieval (format YYYY-MM)
- `period` 1 | 7 | 30, required — Period of time (in days)
- `share_type` 'facebook' — Share Type
- `sourceType` 'nytimes', required
- SourceOkta
- `credentials` union
- object
- `auth_type` 'oauth2.0', required
- `client_id` string, required — The Client ID of your OAuth application.
- `client_secret` string, required — The Client Secret of your OAuth application.
- `refresh_token` string, required — Refresh Token to obtain new Access Token, when it's expired.
- object
- `auth_type` 'oauth2.0_private_key', required
- `client_id` string, required — The Client ID of your OAuth application.
- `key_id` string, required — The key ID (kid).
- `private_key` string, required — The private key in PEM format
- `scope` string, required — The OAuth scope.
- object
- `auth_type` 'api_token', required
- `api_token` string, required — An Okta token. See the docs for instructions on how to generate it.
- `domain` string — The Okta domain. See the docs for instructions on how to find it.
- `start_date` string, date-time — UTC date and time in the format YYYY-MM-DDTHH:MM:SSZ. Any data before this date will not be replicated.
- `sourceType` 'okta', required
- SourceOmnisend
- `api_key` string, required — API Key
- `sourceType` 'omnisend', required
- SourceOnesignal
- `user_auth_key` string, required — OneSignal User Auth Key, see the docs for more information on how to obtain this key.
- `applications` object[], required — Applications keys, see the docs for more information on how to obtain this data
- `app_id` string, required
- `app_name` string
- `app_api_key` string, required
- `start_date` string, date-time, required — The date from which you'd like to replicate data for OneSignal API, in the format YYYY-MM-DDT00:00:00Z. All data generated after this date will be replicated.
- `outcome_names` string, required — Comma-separated list of names and the value (sum/count) for the returned outcome data. See the docs for more details
- `sourceType` 'onesignal', required
- SourceOpenDataDc
- `api_key` string, required
- `location` string — address or place or block
- `marid` string — A unique identifier (Master Address Repository).
- `sourceType` 'open-data-dc', required
- SourceOpenweather
- `appid` string, required — API KEY
- `lang` 'af' | 'al' | 'ar' | 'az' | 'bg' | 'ca' | 'cz' | 'da' | 'de' | 'el' | 'en' | 'eu' | 'fa' | 'fi' | 'fr' | 'gl' | 'he' | 'hi' | 'hr' | 'hu' | 'id' | 'it' | 'ja' | 'kr' | 'la' | 'lt' | 'mk' | 'no' | 'nl' | 'pl' | 'pt' | 'pt_br' | 'ro' | 'ru' | 'sv' | 'se' | 'sk' | 'sl' | 'sp' | 'es' | 'sr' | 'th' | 'tr' | 'ua' | 'uk' | 'vi' | 'zh_cn' | 'zh_tw' | 'zu' — You can use lang parameter to get the output in your language. The contents of the description field will be translated. See here for the list of supported languages.
- `lat` string, required — Latitude, decimal (-90; 90). If you need the geocoder to automatic convert city names and zip-codes to geo coordinates and the other way around, please use the OpenWeather Geocoding API
- `lon` string, required — Longitude, decimal (-180; 180). If you need the geocoder to automatic convert city names and zip-codes to geo coordinates and the other way around, please use the OpenWeather Geocoding API
- `only_current` boolean — True for particular day
- `units` 'standard' | 'metric' | 'imperial' — Units of measurement. standard, metric and imperial units are available. If you do not use the units parameter, standard units will be applied by default.
- `sourceType` 'openweather', required
- SourceOpsgenie
- `api_token` string, required — API token used to access the Opsgenie platform
- `endpoint` string, required — Service endpoint to use for API calls.
- `start_date` string — The date from which you'd like to replicate data from Opsgenie in the format of YYYY-MM-DDT00:00:00Z. All data generated after this date will be replicated. Note that it will be used only in the following incremental streams: issues.
- `sourceType` 'opsgenie', required
- SourceOracle
- `host` string, required — Hostname of the database.
- `port` integer, required — Port of the database. Oracle Corporations recommends the following port numbers: 1521 - Default listening port for client connections to the listener. 2484 - Recommended and officially registered listening port for client connections to the listener using TCP/IP with SSL
- `connection_data` union — Connect data that will be used for DB connection
- object — Use service name
- `connection_type` 'service_name'
- `service_name` string, required
- object — Use SID (Oracle System Identifier)
- `connection_type` 'sid'
- `sid` string, required
- `username` string, required — The username which is used to access the database.
- `password` string — The password associated with the username.
- `schemas` string[] — The list of schemas to sync from. Defaults to user. Case sensitive.
- `jdbc_url_params` string — Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (example: key1=value1&key2=value2&key3=value3).
- `encryption` union — The encryption method with is used when communicating with the database.
- object — Data transfer will not be encrypted.
- `encryption_method` 'unencrypted', required
- object — The native network encryption gives you the ability to encrypt database connections, without the configuration overhead of TCP/IP and SSL/TLS and without the need to open and listen on different ports.
- `encryption_method` 'client_nne', required
- `encryption_algorithm` 'AES256' | 'RC4_56' | '3DES168' — This parameter defines what encryption algorithm is used.
- object — Verify and use the certificate provided by the server.
- `encryption_method` 'encrypted_verify_certificate', required
- `ssl_certificate` string, required — Privacy Enhanced Mail (PEM) files are concatenated certificate containers frequently used in certificate installations.
- `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use.
- object
- `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database
- object
- `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key
- `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel.
- `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections.
- `tunnel_user` string, required — OS-level username for logging into the jump server host.
- `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa )
- object
- `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication
- `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel.
- `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections.
- `tunnel_user` string, required — OS-level username for logging into the jump server host
- `tunnel_user_password` string, required — OS-level password for logging into the jump server host
- `sourceType` 'oracle', required
- SourceOrb
- `api_key` string, required — Orb API Key, issued from the Orb admin console.
- `start_date` string, date-time, required — UTC date and time in the format 2022-03-01T00:00:00Z. Any data with created_at before this data will not be synced. For Subscription Usage, this becomes the `timeframe_start` API parameter.
- `end_date` string — UTC date and time in the format 2022-03-01T00:00:00Z. Any data with created_at after this data will not be synced. For Subscription Usage, this becomes the `timeframe_start` API parameter.
- `lookback_window_days` integer — When set to N, the connector will always refresh resources created within the past N days. By default, updated objects that are not newly created are not incrementally synced.
- `string_event_properties_keys` string[] — Property key names to extract from all events, in order to enrich ledger entries corresponding to an event deduction.
- `numeric_event_properties_keys` string[] — Property key names to extract from all events, in order to enrich ledger entries corresponding to an event deduction.
- `subscription_usage_grouping_key` string — Property key name to group subscription usage by.
- `plan_id` string — Orb Plan ID to filter subscriptions that should have usage fetched.
- `sourceType` 'orb', required
- SourceOrbit
- `api_token` string, required — Authorizes you to work with Orbit workspaces associated with the token.
- `workspace` string, required — The unique name of the workspace that your API token is associated with.
- `start_date` string — Date in the format 2022-06-26. Only load members whose last activities are after this date.
- `sourceType` 'orbit', required
- SourceOura
- `api_key` string, required — API Key
- `start_datetime` string, date-time — Start datetime to sync from. Default is current UTC datetime minus 1 day.
- `end_datetime` string, date-time — End datetime to sync until. Default is current UTC datetime.
- `sourceType` 'oura', required
- SourceOutbrainAmplify
- `credentials` union, required — Credentials for making authenticated requests requires either username/password or access_token.
- object
- `type` 'access_token', required
- `access_token` string, required — Access Token for making authenticated requests.
- object
- `type` 'username_password', required
- `username` string, required — Add Username for authentication.
- `password` string, required — Add Password for authentication.
- `report_granularity` 'daily' | 'weekly' | 'monthly' — The granularity used for periodic data in reports. See the docs.
- `geo_location_breakdown` 'country' | 'region' | 'subregion' — The granularity used for geo location data in reports.
- `start_date` string, required — Date in the format YYYY-MM-DD eg. 2017-01-25. Any data before this date will not be replicated.
- `end_date` string — Date in the format YYYY-MM-DD.
- `sourceType` 'outbrain-amplify', required
- SourceOutreach
- `client_id` string, required — The Client ID of your Outreach developer application.
- `client_secret` string, required — The Client Secret of your Outreach developer application.
- `refresh_token` string, required — The token for obtaining the new access token.
- … truncated; see the full OpenAPI document linked below
## Response `200`
Update a Source
- SourceResponse — Provides details of a single source.
- `sourceId` string, UUID, required
- `name` string, required
- `sourceType` string, required
- `workspaceId` string, UUID, required
- `configuration` union, required — The values required to configure the source.
- SourceAha
- `api_key` string, required — API Key
- `url` string, required — URL
- `sourceType` 'aha', required
- Source7shifts
- `access_token` string, required — Access token to use for authentication. Generate it in the 7shifts Developer Tools.
- `start_date` string, date-time, required
- `sourceType` '7shifts', required
- SourceActivecampaign
- `api_key` string, required — API Key
- `account_username` string, required — Account Username
- `sourceType` 'activecampaign', required
- SourceAirbyte
- `client_id` string, required
- `start_date` string, date-time, required
- `client_secret` string, required
- `sourceType` 'airbyte', required
- SourceAircall
- `api_id` string, required — App ID found at settings https://dashboard.aircall.io/integrations/api-keys
- `api_token` string, required — App token found at settings (Ref- https://dashboard.aircall.io/integrations/api-keys)
- `start_date` string, date-time, required — Date time filter for incremental filter, Specify which date to extract from.
- `sourceType` 'aircall', required
- SourceAirtable
- `credentials` union
- object
- `auth_method` 'oauth2.0'
- `client_id` string, required — The client ID of the Airtable developer application.
- `client_secret` string, required — The client secret the Airtable developer application.
- `access_token` string — Access Token for making authenticated requests.
- `token_expiry_date` string, date-time — The date-time when the access token should be refreshed.
- `refresh_token` string, required — The key to refresh the expired access token.
- object
- `auth_method` 'api_key'
- `api_key` string, required — The Personal Access Token for the Airtable account. See the Support Guide for more information on how to obtain this token.
- `sourceType` 'airtable'
- SourceAlgolia
- `api_key` string, required
- `application_id` string, required — The application ID for your application found in settings
- `search_query` string — Search query to be used with indexes_query stream with format defined in `https://www.algolia.com/doc/rest-api/search/#tag/Search/operation/searchSingleIndex`
- `start_date` string, date-time, required
- `object_id` string — Object ID within index for search queries
- `sourceType` 'algolia', required
- SourceAmazonAds
- `auth_type` 'oauth2.0'
- `client_id` string, required — The client ID of your Amazon Ads developer application. See the docs for more information.
- `client_secret` string, required — The client secret of your Amazon Ads developer application. See the docs for more information.
- `refresh_token` string, required — Amazon Ads refresh token. See the docs for more information on how to obtain this token.
- `region` 'NA' | 'EU' | 'FE' — Region to pull data from (EU/NA/FE). See docs for more details.
- `start_date` string, date — The Start date for collecting reports, should not be more than 60 days in the past. In YYYY-MM-DD format
- `profiles` integer[] — Profile IDs you want to fetch data for. The Amazon Ads source connector supports only profiles with seller and vendor type, profiles with agency type will be ignored. See docs for more details. Note: If Marketplace IDs are also selected, profiles will be selected if they match the Profile ID OR the Marketplace ID.
- `marketplace_ids` string[] — Marketplace IDs you want to fetch data for. Note: If Profile IDs are also selected, profiles will be selected if they match the Profile ID OR the Marketplace ID.
- `state_filter` string[] — Reflects the state of the Display, Product, and Brand Campaign streams as enabled, paused, or archived. If you do not populate this field, it will be ignored completely.
- `look_back_window` integer — The amount of days to go back in time to get the updated data from Amazon Ads
- `report_record_types` string[] — Optional configuration which accepts an array of string of record types. Leave blank for default behaviour to pull all report types. Use this config option only if you want to pull specific report type(s). See docs for more details
- `sourceType` 'amazon-ads', required
- SourceAmazonSellerPartner
- `auth_type` 'oauth2.0'
- `aws_environment` 'PRODUCTION' | 'SANDBOX', required — Select the AWS Environment.
- `region` 'AE' | 'AU' | 'BE' | 'BR' | 'CA' | 'DE' | 'EG' | 'ES' | 'FR' | 'GB' | 'IN' | 'IT' | 'JP' | 'MX' | 'NL' | 'PL' | 'SA' | 'SE' | 'SG' | 'TR' | 'UK' | 'US', required — Select the AWS Region.
- `account_type` 'Seller' | 'Vendor', required — Type of the Account you're going to authorize the Airbyte application by
- `lwa_app_id` string, required — Your Login with Amazon Client ID.
- `lwa_client_secret` string, required — Your Login with Amazon Client Secret.
- `refresh_token` string, required — The Refresh Token obtained via OAuth flow authorization.
- `replication_start_date` string, date-time — UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated. If start date is not provided or older than 2 years ago from today, the date 2 years ago from today will be used.
- `replication_end_date` string, date-time — UTC date and time in the format 2017-01-25T00:00:00Z. Any data after this date will not be replicated.
- `period_in_days` integer — For syncs spanning a large date range, this option is used to request data in a smaller fixed window to improve sync reliability. This time window can be configured granularly by day.
- `report_options_list` object[] — Additional information passed to reports. This varies by report type.
- `report_name` 'GET_AFN_INVENTORY_DATA' | 'GET_AFN_INVENTORY_DATA_BY_COUNTRY' | 'GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL' | 'GET_FBA_ESTIMATED_FBA_FEES_TXT_DATA' | 'GET_FBA_FULFILLMENT_CUSTOMER_RETURNS_DATA' | 'GET_FBA_FULFILLMENT_CUSTOMER_SHIPMENT_PROMOTION_DATA' | 'GET_FBA_FULFILLMENT_CUSTOMER_SHIPMENT_REPLACEMENT_DATA' | 'GET_FBA_FULFILLMENT_REMOVAL_ORDER_DETAIL_DATA' | 'GET_FBA_FULFILLMENT_REMOVAL_SHIPMENT_DETAIL_DATA' | 'GET_FBA_INVENTORY_PLANNING_DATA' | 'GET_FBA_MYI_UNSUPPRESSED_INVENTORY_DATA' | 'GET_FBA_REIMBURSEMENTS_DATA' | 'GET_FBA_SNS_FORECAST_DATA' | 'GET_FBA_SNS_PERFORMANCE_DATA' | 'GET_FBA_STORAGE_FEE_CHARGES_DATA' | 'GET_FLAT_FILE_ACTIONABLE_ORDER_DATA_SHIPPING' | 'GET_FLAT_FILE_ALL_ORDERS_DATA_BY_LAST_UPDATE_GENERAL' | 'GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL' | 'GET_FLAT_FILE_ARCHIVED_ORDERS_DATA_BY_ORDER_DATE' | 'GET_FLAT_FILE_OPEN_LISTINGS_DATA' | 'GET_FLAT_FILE_RETURNS_DATA_BY_RETURN_DATE' | 'GET_LEDGER_DETAIL_VIEW_DATA' | 'GET_LEDGER_SUMMARY_VIEW_DATA' | 'GET_MERCHANT_CANCELLED_LISTINGS_DATA' | 'GET_MERCHANT_LISTINGS_ALL_DATA' | 'GET_MERCHANT_LISTINGS_DATA' | 'GET_MERCHANT_LISTINGS_DATA_BACK_COMPAT' | 'GET_MERCHANT_LISTINGS_INACTIVE_DATA' | 'GET_MERCHANTS_LISTINGS_FYP_REPORT' | 'GET_ORDER_REPORT_DATA_SHIPPING' | 'GET_RESTOCK_INVENTORY_RECOMMENDATIONS_REPORT' | 'GET_SELLER_FEEDBACK_DATA' | 'GET_STRANDED_INVENTORY_UI_DATA' | 'GET_V2_SETTLEMENT_REPORT_DATA_FLAT_FILE' | 'GET_XML_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL' | 'GET_XML_BROWSE_TREE_DATA' | 'GET_VENDOR_REAL_TIME_INVENTORY_REPORT' | 'GET_BRAND_ANALYTICS_MARKET_BASKET_REPORT' | 'GET_BRAND_ANALYTICS_SEARCH_TERMS_REPORT' | 'GET_BRAND_ANALYTICS_REPEAT_PURCHASE_REPORT' | 'GET_SALES_AND_TRAFFIC_REPORT' | 'GET_VENDOR_SALES_REPORT' | 'GET_VENDOR_INVENTORY_REPORT' | 'GET_VENDOR_NET_PURE_PRODUCT_MARGIN_REPORT' | 'GET_VENDOR_TRAFFIC_REPORT', required
- `stream_name` string, required
- `options_list` object[], required — List of options
- `option_name` string, required
- `option_value` string, required
- `wait_to_avoid_fatal_errors` boolean — For report based streams with known amount of requests per time period, this option will use waiting time between requests to avoid fatal statuses in reports. See Troubleshooting section for more details
- `sourceType` 'amazon-seller-partner', required
- SourceAmazonSqs
- `queue_url` string, required — URL of the SQS Queue
- `region` 'af-south-1' | 'ap-east-1' | 'ap-northeast-1' | 'ap-northeast-2' | 'ap-northeast-3' | 'ap-south-1' | 'ap-south-2' | 'ap-southeast-1' | 'ap-southeast-2' | 'ap-southeast-3' | 'ap-southeast-4' | 'ca-central-1' | 'ca-west-1' | 'cn-north-1' | 'cn-northwest-1' | 'eu-central-1' | 'eu-central-2' | 'eu-north-1' | 'eu-south-1' | 'eu-south-2' | 'eu-west-1' | 'eu-west-2' | 'eu-west-3' | 'il-central-1' | 'me-central-1' | 'me-south-1' | 'sa-east-1' | 'us-east-1' | 'us-east-2' | 'us-gov-east-1' | 'us-gov-west-1' | 'us-west-1' | 'us-west-2', required — AWS Region of the SQS Queue
- `delete_messages` boolean, required — If Enabled, messages will be deleted from the SQS Queue after being read. If Disabled, messages are left in the queue and can be read more than once. WARNING: Enabling this option can result in data loss in cases of failure, use with caution, see documentation for more detail.
- `max_batch_size` integer — Max amount of messages to get in one batch (10 max)
- `max_wait_time` integer — Max amount of time in seconds to wait for messages in a single poll (20 max)
- `attributes_to_return` string — Comma separated list of Mesage Attribute names to return
- `visibility_timeout` integer — Modify the Visibility Timeout of the individual message from the Queue's default (seconds).
- `access_key` string — The Access Key ID of the AWS IAM Role to use for pulling messages
- `secret_key` string — The Secret Key of the AWS IAM Role to use for pulling messages
- `sourceType` 'amazon-sqs', required
- SourceAmplitude
- `data_region` 'Standard Server' | 'EU Residency Server' — Amplitude data region server
- `api_key` string, required — Amplitude API Key. See the setup guide for more information on how to obtain this key.
- `secret_key` string, required — Amplitude Secret Key. See the setup guide for more information on how to obtain this key.
- `start_date` string, date-time, required — UTC date and time in the format 2021-01-25T00:00:00Z. Any data before this date will not be replicated.
- `request_time_range` integer — According to Considerations too big time range in request can cause a timeout error. In this case, set shorter time interval in hours.
- `active_users_group_by_country` boolean — According to Considerations the grouping by `Country` is optional, if you're facing issues fetching the stream, or checking the connection please set this to `False` instead.
- `sourceType` 'amplitude', required
- SourceApifyDataset
- `token` string, required — Personal API token of your Apify account. In Apify Console, you can find your API token in the Settings section under the Integrations tab after you login. See the Apify Docs for more information.
- `dataset_id` string, required — ID of the dataset you would like to load to Airbyte. In Apify Console, you can view your datasets in the Storage section under the Datasets tab after you login. See the Apify Docs for more information.
- `sourceType` 'apify-dataset', required
- SourceAppcues
- `username` string, required
- `password` string
- `account_id` string, required — Account ID of Appcues found in account settings page (https://studio.appcues.com/settings/account)
- `start_date` string, date-time, required
- `sourceType` 'appcues', required
- SourceAppfigures
- `api_key` string, required
- `search_store` string — The store which needs to be searched in streams
- `start_date` string, date-time, required
- `group_by` 'network' | 'product' | 'country' | 'date' — Category term for grouping the search results
- `sourceType` 'appfigures', required
- SourceAppfollow
- `api_secret` string — API Key provided by Appfollow
- `sourceType` 'appfollow', required
- SourceAppleSearchAds
- `org_id` integer, required — The identifier of the organization that owns the campaign. Your Org Id is the same as your account in the Apple Search Ads UI.
- `end_date` string — Data is retrieved until that date (included)
- `client_id` string, required — A user identifier for the token request. See here
- `start_date` string, required — Start getting data from that date.
- `client_secret` string, required — A string that authenticates the user’s setup request. See here
- `sourceType` 'apple-search-ads', required
- SourceAsana
- `credentials` union — Choose how to authenticate to Github
- object
- `option_title` 'OAuth Credentials' — OAuth Credentials
- `client_id` string, required
- `client_secret` string, required
- `refresh_token` string, required
- object
- `option_title` 'PAT Credentials' — PAT Credentials
- `personal_access_token` string, required — Asana Personal Access Token (generate yours here).
- `organization_export_ids` unknown[] — Globally unique identifiers for the organization exports
- unknown
- `sourceType` 'asana'
- SourceAshby
- `api_key` string, required — The Ashby API Key, see doc here.
- `start_date` string, required — UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.
- `sourceType` 'ashby', required
- SourceAuth0
- `base_url` string, required — The Authentication API is served over HTTPS. All URLs referenced in the documentation have the following base `https://YOUR_DOMAIN`
- `credentials` union, required
- object
- `auth_type` 'oauth2_confidential_application', required
- `client_id` string, required — Your application's Client ID. You can find this value on the application's settings tab after you login the admin portal.
- `client_secret` string, required — Your application's Client Secret. You can find this value on the application's settings tab after you login the admin portal.
- `audience` string, required — The audience for the token, which is your API. You can find this in the Identifier field on your API's settings tab
- object
- `auth_type` 'oauth2_access_token', required
- `access_token` string, required — Also called API Access Token The access token used to call the Auth0 Management API Token. It's a JWT that contains specific grant permissions knowns as scopes.
- `start_date` string — UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.
- `sourceType` 'auth0', required
- SourceAwsCloudtrail
- `aws_key_id` string, required — AWS CloudTrail Access Key ID. See the docs for more information on how to obtain this key.
- `aws_secret_key` string, required — AWS CloudTrail Access Key ID. See the docs for more information on how to obtain this key.
- `aws_region_name` string, required — The default AWS Region to use, for example, us-west-1 or us-west-2. When specifying a Region inline during client initialization, this property is named region_name.
- `start_date` string, date — The date you would like to replicate data. Data in AWS CloudTrail is available for last 90 days only. Format: YYYY-MM-DD.
- `lookup_attributes_filter` object
- `attribute_key` string, required
- `attribute_value` string, required
- `sourceType` 'aws-cloudtrail', required
- SourceAzureBlobStorage — NOTE: When this Spec is changed, legacy_config_transformer.py must also be modified to uptake the changes because it is responsible for converting legacy Azure Blob Storage v0 configs into v1 configs using the File-Based CDK.
- `start_date` string, date-time — UTC date and time in the format 2017-01-25T00:00:00.000000Z. Any file modified before this date will not be replicated.
- `streams` object[], required — Each instance of this configuration defines a stream. Use this to define which files belong in the stream, their format, and how they should be parsed and validated. When sending data to warehouse destination such as Snowflake or BigQuery, each stream is a separate table.
- `name` string, required — The name of the stream.
- `globs` string[] — The pattern used to specify which files should be selected from the file system. For more information on glob pattern matching look here.
- `validation_policy` 'Emit Record' | 'Skip Record' | 'Wait for Discover' — The name of the validation policy that dictates sync behavior when a record does not adhere to the stream schema.
- `input_schema` string — The schema that will be used to validate records extracted from the file. This will override the stream schema that is auto-detected from incoming files.
- `days_to_sync_if_history_is_full` integer — When the state history of the file store is full, syncs will only read files that were last modified in the provided day range.
- `format` union, required — The configuration options that are used to alter how to read incoming files that deviate from the standard formatting.
- object
- `filetype` 'avro', required
- `double_as_string` boolean — Whether to convert double fields to strings. This is recommended if you have decimal numbers with a high degree of precision because there can be a loss precision when handling floating point numbers.
- object
- `filetype` 'csv', required
- `delimiter` string — The character delimiting individual cells in the CSV data. This may only be a 1-character string. For tab-delimited data enter '\t'.
- `quote_char` string — The character used for quoting CSV values. To disallow quoting, make this field blank.
- `escape_char` string — The character used for escaping special characters. To disallow escaping, leave this field blank.
- `encoding` string — The character encoding of the CSV data. Leave blank to default to UTF8. See list of python encodings for allowable options.
- `double_quote` boolean — Whether two quotes in a quoted CSV value denote a single quote in the data.
- `null_values` string[] — A set of case-sensitive strings that should be interpreted as null values. For example, if the value 'NA' should be interpreted as null, enter 'NA' in this field.
- `strings_can_be_null` boolean — Whether strings can be interpreted as null values. If true, strings that match the null_values set will be interpreted as null. If false, strings that match the null_values set will be interpreted as the string itself.
- `skip_rows_before_header` integer — The number of rows to skip before the header row. For example, if the header row is on the 3rd row, enter 2 in this field.
- `skip_rows_after_header` integer — The number of rows to skip after the header row.
- `header_definition` union — How headers will be defined. `User Provided` assumes the CSV does not have a header row and uses the headers provided and `Autogenerated` assumes the CSV does not have a header row and the CDK will generate headers using for `f{i}` where `i` is the index starting from 0. Else, the default behavior is to use the header from the CSV file. If a user wants to autogenerate or provide column names for a CSV having headers, they can skip rows.
- …
- `true_values` string[] — A set of case-sensitive strings that should be interpreted as true values.
- `false_values` string[] — A set of case-sensitive strings that should be interpreted as false values.
- `ignore_errors_on_fields_mismatch` boolean — Whether to ignore errors that occur when the number of fields in the CSV does not match the number of columns in the schema.
- object
- `filetype` 'jsonl', required
- object
- `filetype` 'parquet', required
- `decimal_as_float` boolean — Whether to convert decimal fields to floats. There is a loss of precision when converting decimals to floats, so this is not recommended.
- object — Extract text from document formats (.pdf, .docx, .md, .pptx) and emit as one record per file.
- `filetype` 'unstructured', required
- `skip_unprocessable_files` boolean — If true, skip files that cannot be parsed and pass the error message along as the _ab_source_file_parse_error field. If false, fail the sync.
- `strategy` 'auto' | 'fast' | 'ocr_only' | 'hi_res' — The strategy used to parse documents. `fast` extracts text directly from the document which doesn't work for all files. `ocr_only` is more reliable, but slower. `hi_res` is the most reliable, but requires an API key and a hosted instance of unstructured and can't be used with local mode. See the unstructured.io documentation for more details: https://unstructured-io.github.io/unstructured/core/partition.html#partition-pdf
- `processing` object — Process files locally, supporting `fast` and `ocr` modes. This is the default option.
- …
- `schemaless` boolean — When enabled, syncs will not validate or structure records against the stream's schema.
- `credentials` union, required — Credentials for connecting to the Azure Blob Storage
- object
- `auth_type` 'oauth2', required
- `tenant_id` string, required — Tenant ID of the Microsoft Azure Application user
- `client_id` string, required — Client ID of your Microsoft developer application
- `client_secret` string, required — Client Secret of your Microsoft developer application
- `refresh_token` string, required — Refresh Token of your Microsoft developer application
- object
- `auth_type` 'storage_account_key', required
- `azure_blob_storage_account_key` string, required — The Azure blob storage account key.
- `azure_blob_storage_account_name` string, required — The account's name of the Azure Blob Storage.
- `azure_blob_storage_container_name` string, required — The name of the Azure blob storage container.
- `azure_blob_storage_endpoint` string — This is Azure Blob Storage endpoint domain name. Leave default value (or leave it empty if run container from command line) to use Microsoft native from example.
- `sourceType` 'azure-blob-storage', required
- SourceAzureTable
- `storage_account_name` string, required — The name of your storage account.
- `storage_access_key` string, required — Azure Table Storage Access Key. See the docs for more information on how to obtain this key.
- `storage_endpoint_suffix` string — Azure Table Storage service account URL suffix. See the docs for more information on how to obtain endpoint suffix
- `sourceType` 'azure-table', required
- SourceBambooHr
- `api_key` string, required — Api key of bamboo hr
- `subdomain` string, required — Sub Domain of bamboo hr
- `custom_reports_fields` string — Comma-separated list of fields to include in custom reports.
- `custom_reports_include_default_fields` boolean — If true, the custom reports endpoint will include the default fields defined here: https://documentation.bamboohr.com/docs/list-of-field-names.
- `start_date` string, date-time
- `sourceType` 'bamboo-hr', required
- SourceBasecamp
- `account_id` number, required
- `start_date` string, date-time, required
- `client_id` string, required
- `client_secret` string, required
- `client_refresh_token_2` string, required
- `sourceType` 'basecamp', required
- SourceBeamer
- `api_key` string, required
- `start_date` string, date-time, required
- `sourceType` 'beamer', required
- SourceBigquery
- `project_id` string, required — The GCP project ID for the project containing the target BigQuery dataset.
- `dataset_id` string — The dataset ID to search for tables and views. If you are only loading data from one dataset, setting this option could result in much faster schema discovery.
- `credentials_json` string, required — The contents of your Service Account Key JSON file. See the docs for more information on how to obtain this key.
- `sourceType` 'bigquery', required
- SourceBingAds
- `auth_method` 'oauth2.0'
- `tenant_id` string — The Tenant ID of your Microsoft Advertising developer application. Set this to "common" unless you know you need a different value.
- `client_id` string, required — The Client ID of your Microsoft Advertising developer application.
- `client_secret` string — The Client Secret of your Microsoft Advertising developer application.
- `refresh_token` string, required — Refresh Token to renew the expired Access Token.
- `developer_token` string, required — Developer token associated with user. See more info in the docs.
- `account_names` object[] — Predicates that will be used to sync data by specific accounts.
- `operator` 'Contains' | 'Equals', required — An Operator that will be used to filter accounts. The Contains predicate has features for matching words, matching inflectional forms of words, searching using wildcard characters, and searching using proximity. The Equals is used to return all rows where account name is equal(=) to the string that you provided
- `name` string, required — Account Name is a string value for comparing with the specified predicate.
- `reports_start_date` string, date — The start date from which to begin replicating report data. Any data generated before this date will not be replicated in reports. This is a UTC date in YYYY-MM-DD format. If not set, data from previous and current calendar year will be replicated.
- `lookback_window` integer — Also known as attribution or conversion window. How far into the past to look for records (in days). If your conversion window has an hours/minutes granularity, round it up to the number of days exceeding. Used only for performance report streams in incremental mode without specified Reports Start Date.
- `custom_reports` object[] — You can add your Custom Bing Ads report by creating one.
- `name` string, required — The name of the custom report, this name would be used as stream name
- `reporting_object` 'AccountPerformanceReportRequest' | 'AdDynamicTextPerformanceReportRequest' | 'AdExtensionByAdReportRequest' | 'AdExtensionByKeywordReportRequest' | 'AdExtensionDetailReportRequest' | 'AdGroupPerformanceReportRequest' | 'AdPerformanceReportRequest' | 'AgeGenderAudienceReportRequest' | 'AudiencePerformanceReportRequest' | 'CallDetailReportRequest' | 'CampaignPerformanceReportRequest' | 'ConversionPerformanceReportRequest' | 'DestinationUrlPerformanceReportRequest' | 'DSAAutoTargetPerformanceReportRequest' | 'DSACategoryPerformanceReportRequest' | 'DSASearchQueryPerformanceReportRequest' | 'GeographicPerformanceReportRequest' | 'GoalsAndFunnelsReportRequest' | 'HotelDimensionPerformanceReportRequest' | 'HotelGroupPerformanceReportRequest' | 'KeywordPerformanceReportRequest' | 'NegativeKeywordConflictReportRequest' | 'ProductDimensionPerformanceReportRequest' | 'ProductMatchCountReportRequest' | 'ProductNegativeKeywordConflictReportRequest' | 'ProductPartitionPerformanceReportRequest' | 'ProductPartitionUnitPerformanceReportRequest' | 'ProductSearchQueryPerformanceReportRequest' | 'ProfessionalDemographicsAudienceReportRequest' | 'PublisherUsagePerformanceReportRequest' | 'SearchCampaignChangeHistoryReportRequest' | 'SearchQueryPerformanceReportRequest' | 'ShareOfVoiceReportRequest' | 'UserLocationPerformanceReportRequest', required — The name of the the object derives from the ReportRequest object. You can find it in Bing Ads Api docs - Reporting API - Reporting Data Objects.
- `report_columns` string[], required — A list of available report object columns. You can find it in description of reporting object that you want to add to custom report.
- `report_aggregation` string, required — A list of available aggregations.
- `sourceType` 'bing-ads', required
- SourceBitly
- `api_key` string, required
- `start_date` string, date-time, required
- `end_date` string, date-time, required
- `sourceType` 'bitly', required
- SourceBraintree
- `merchant_id` string, required — The unique identifier for your entire gateway account. See the docs for more information on how to obtain this ID.
- `public_key` string, required — Braintree Public Key. See the docs for more information on how to obtain this key.
- `private_key` string, required — Braintree Private Key. See the docs for more information on how to obtain this key.
- `start_date` string, date-time — UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.
- `environment` 'Development' | 'Sandbox' | 'Qa' | 'Production', required — Environment specifies where the data will come from.
- `sourceType` 'braintree', required
- SourceBraze
- `url` string, required — Braze REST API endpoint
- `api_key` string, required — Braze REST API key
- `start_date` string, date, required — Rows after this date will be synced
- `sourceType` 'braze', required
- SourceBreezyHr
- `api_key` string, required
- `company_id` string, required
- `sourceType` 'breezy-hr', required
- SourceBrevo
- `api_key` string, required
- `start_date` string, date-time, required
- `sourceType` 'brevo', required
- SourceBugsnag
- `auth_token` string, required — Personal auth token for accessing the Bugsnag API. Generate it in the My Account section of Bugsnag settings.
- `start_date` string, date-time, required
- `sourceType` 'bugsnag', required
- SourceBuildkite
- `api_key` string, required
- `start_date` string, date-time, required
- `sourceType` 'buildkite', required
- SourceBuzzsprout
- `api_key` string, required
- `podcast_id` string, required — Podcast ID found in `https://www.buzzsprout.com/my/profile/api`
- `start_date` string, date-time, required
- `sourceType` 'buzzsprout', required
- SourceCalendly
- `api_key` string, required — Go to Integrations → API & Webhooks to obtain your bearer token. https://calendly.com/integrations/api_webhooks
- `start_date` string, date-time, required
- `sourceType` 'calendly', required
- SourceCallrail
- `api_key` string, required — API access key
- `account_id` string, required — Account ID
- `start_date` string, required — Start getting data from that date.
- `sourceType` 'callrail', required
- SourceCampaignMonitor
- `username` string, required
- `password` string
- `start_date` string — Date from when the sync should start
- `sourceType` 'campaign-monitor', required
- SourceCanny
- `api_key` string, required — You can find your secret API key in Your Canny Subdomain > Settings > API
- `sourceType` 'canny', required
- SourceCareQualityCommission
- `api_key` string, required — Your CQC Primary Key. See https://www.cqc.org.uk/about-us/transparency/using-cqc-data#api for steps to generate one.
- `sourceType` 'care-quality-commission', required
- SourceCart
- `credentials` union
- object
- `auth_type` 'CENTRAL_API_ROUTER', required
- `user_name` string, required — Enter your application's User Name
- `user_secret` string, required — Enter your application's User Secret
- `site_id` string, required — You can determine a site provisioning site Id by hitting https://site.com/store/sitemonitor.aspx and reading the response param PSID
- object
- `auth_type` 'SINGLE_STORE_ACCESS_TOKEN', required
- `access_token` string, required — Access Token for making authenticated requests.
- `store_name` string, required — The name of Cart.com Online Store. All API URLs start with https://[mystorename.com]/api/v1/, where [mystorename.com] is the domain name of your store.
- `start_date` string, required — The date from which you'd like to replicate the data
- `sourceType` 'cart', required
- SourceChameleon
- `api_key` string, required
- `start_date` string, date-time, required
- `limit` string — Max records per page limit
- `filter` 'tour' | 'survey' | 'launcher' — Filter for using in the `segments_experiences` stream
- `end_date` string, date-time — End date for incremental sync
- `sourceType` 'chameleon', required
- SourceChargebee
- `site_api_key` string, required — Chargebee API Key. See the docs for more information on how to obtain this key.
- `site` string, required — The site prefix for your Chargebee instance.
- `start_date` string, date-time, required — UTC date and time in the format 2017-01-25T00:00:00.000Z. Any data before this date will not be replicated.
- `product_catalog` '1.0' | '2.0' — Product Catalog version of your Chargebee site. Instructions on how to find your version you may find here under `API Version` section. If left blank, the product catalog version will be set to 2.0.
- `sourceType` 'chargebee', required
- SourceChargify
- `api_key` string, required — Maxio Advanced Billing/Chargify API Key.
- `domain` string, required — Chargify domain. Normally this domain follows the following format
- `username` string, required
- `password` string
- `sourceType` 'chargify', required
- SourceChartmogul
- `api_key` string, required — Your Chartmogul API key. See the docs for info on how to obtain this.
- `start_date` string, date-time, required — UTC date and time in the format 2017-01-25T00:00:00Z. When feasible, any data before this date will not be replicated.
- `sourceType` 'chartmogul', required
- SourceCimis
- `api_key` string, required
- `targets_type` 'WSN station numbers' | 'California zip codes' | 'decimal-degree coordinates' | 'street addresses', required
- `targets` unknown[], required
- unknown
- `daily_data_items` unknown[]
- unknown
- `hourly_data_items` unknown[]
- unknown
- `start_date` string, date-time, required
- `end_date` string, date-time, required
- `unit_of_measure` 'E' | 'M'
- `sourceType` 'cimis', required
- SourceCircleci
- `api_key` string, required
- `org_id` string, required — The org ID found in `https://app.circleci.com/settings/organization/circleci/xxxxx/overview`
- `start_date` string, date-time, required
- `project_id` string, required — Project ID found in the project settings
- `job_id` string — Job ID for fetching information
- `workflow_id` string — workflow ID of a project pipeline
- `workflow_name` string — Workflow name for fetching information
- `job_number` string — Job Number of the workflow
- `sourceType` 'circleci', required
- SourceClazar
- `client_id` string, required
- `client_secret` string, required
- `sourceType` 'clazar', required
- SourceClickhouse
- `host` string, required — The host endpoint of the Clickhouse cluster.
- `port` integer, required — The port of the database.
- `database` string, required — The name of the database.
- `username` string, required — The username which is used to access the database.
- `password` string — The password associated with this username.
- `jdbc_url_params` string — Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (Eg. key1=value1&key2=value2&key3=value3). For more information read about JDBC URL parameters.
- `ssl` boolean — Encrypt data using SSL.
- `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use.
- object
- `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database
- object
- `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key
- `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel.
- `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections.
- `tunnel_user` string, required — OS-level username for logging into the jump server host.
- `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa )
- object
- `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication
- `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel.
- `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections.
- `tunnel_user` string, required — OS-level username for logging into the jump server host
- `tunnel_user_password` string, required — OS-level password for logging into the jump server host
- `sourceType` 'clickhouse', required
- SourceClickupApi
- `api_token` string, required — Every ClickUp API call required authentication. This field is your personal API token. See here.
- `include_closed_tasks` boolean — Include or exclude closed tasks. By default, they are excluded. See here.
- `sourceType` 'clickup-api', required
- SourceClockify
- `api_key` string, required — You can get your api access_key here This API is Case Sensitive.
- `api_url` string — The URL for the Clockify API. This should only need to be modified if connecting to an enterprise version of Clockify.
- `workspace_id` string, required — WorkSpace Id
- `sourceType` 'clockify', required
- SourceCloseCom
- `api_key` string, required — Close.com API key (usually starts with 'api_'; find yours here).
- `start_date` string, date — The start date to sync data; all data after this date will be replicated. Leave blank to retrieve all the data available in the account. Format: YYYY-MM-DD.
- `sourceType` 'close-com', required
- SourceCoassemble
- `user_token` string, required
- `user_id` string, required
- `sourceType` 'coassemble', required
- SourceCoda
- `auth_token` string, required — Bearer token
- `sourceType` 'coda', required
- SourceCoinApi
- `api_key` string, required — API Key
- `environment` 'sandbox' | 'production', required — The environment to use. Either sandbox or production.
- `symbol_id` string, required — The symbol ID to use. See the documentation for a list. https://docs.coinapi.io/#list-all-symbols-get
- `period` string, required — The period to use. See the documentation for a list. https://docs.coinapi.io/#list-all-periods-get
- `start_date` string, required — The start date in ISO 8601 format.
- `end_date` string — The end date in ISO 8601 format. If not supplied, data will be returned from the start date to the current time, or when the count of result elements reaches its limit.
- `limit` integer — The maximum number of elements to return. If not supplied, the default is 100. For numbers larger than 100, each 100 items is counted as one request for pricing purposes. Maximum value is 100000.
- `sourceType` 'coin-api', required
- SourceCoinmarketcap
- `api_key` string, required — Your API Key. See here. The token is case sensitive.
- `data_type` 'latest' | 'historical', required — /latest: Latest market ticker quotes and averages for cryptocurrencies and exchanges. /historical: Intervals of historic market data like OHLCV data or data for use in charting libraries. See here.
- `symbols` string[] — Cryptocurrency symbols. (only used for quotes stream)
- `sourceType` 'coinmarketcap', required
- SourceConfigcat
- `username` string, required — Basic auth user name. See here.
- `password` string, required — Basic auth password. See here.
- `sourceType` 'configcat', required
- SourceConfluence
- `email` string, required — Your Confluence login email
- `api_token` string, required — Please follow the Jira confluence for generating an API token: generating an API token.
- `domain_name` string, required — Your Confluence domain name
- `sourceType` 'confluence', required
- SourceConvertkit
- `api_secret` string, required — API Secret
- `sourceType` 'convertkit', required
- SourceConvex
- `deployment_url` string, required
- `access_key` string, required — API access key used to retrieve data from Convex.
- `sourceType` 'convex', required
- SourceCopper
- `api_key` string, required — Copper API key
- `user_email` string, required — user email used to login in to Copper
- `sourceType` 'copper', required
- SourceCountercyclical
- `api_key` string, required
- `sourceType` 'countercyclical', required
- SourceCustomerIo
- `app_api_key` string, required
- `sourceType` 'customer-io', required
- SourceDatadog
- `api_key` string, required — Datadog API key
- `application_key` string, required — Datadog application key
- `query` string — The search query. This just applies to Incremental syncs. If empty, it'll collect all logs.
- `start_date` string — UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated. This just applies to Incremental syncs.
- `site` 'datadoghq.com' | 'us3.datadoghq.com' | 'us5.datadoghq.com' | 'datadoghq.eu' | 'ddog-gov.com' — The site where Datadog data resides in.
- `end_date` string — UTC date and time in the format 2017-01-25T00:00:00Z. Data after this date will not be replicated. An empty value will represent the current datetime for each execution. This just applies to Incremental syncs.
- `max_records_per_request` integer — Maximum number of records to collect per request.
- `queries` object[] — List of queries to be run and used as inputs.
- `name` string, required — The variable name for use in queries.
- `query` string, required — A classic query string.
- `data_source` 'metrics' | 'cloud_cost' | 'logs' | 'rum', required — A data source that is powered by the platform.
- `sourceType` 'datadog', required
- SourceDatascope
- `api_key` string, required — API Key
- `start_date` string, required — Start date for the data to be replicated
- `sourceType` 'datascope', required
- SourceDbt
- `api_key_2` string, required
- `account_id` string, required
- `sourceType` 'dbt', required
- SourceDelighted
- `api_key` string, required — A Delighted API key.
- `since` string, date-time, required — The date from which you'd like to replicate the data
- `sourceType` 'delighted', required
- SourceDixa
- `api_token` string, required — Dixa API token
- `batch_size` integer — Number of days to batch into one request. Max 31.
- `start_date` string, date-time, required — The connector pulls records updated from this date onwards.
- `sourceType` 'dixa', required
- SourceDockerhub
- `docker_username` string, required — Username of DockerHub person or organization (for https://hub.docker.com/v2/repositories/USERNAME/ API call)
- `sourceType` 'dockerhub', required
- SourceDremio
- `api_key` string, required — API Key that is generated when you authenticate to Dremio API
- `base_url` string, required — URL of your Dremio instance
- `sourceType` 'dremio', required
- SourceDrift
- `credentials` union
- object
- `credentials` 'oauth2.0'
- `client_id` string, required — The Client ID of your Drift developer application.
- `client_secret` string, required — The Client Secret of your Drift developer application.
- `access_token` string, required — Access Token for making authenticated requests.
- `refresh_token` string, required — Refresh Token to renew the expired Access Token.
- object
- `credentials` 'access_token'
- `access_token` string, required — Drift Access Token. See the docs for more information on how to generate this key.
- `email` string — Email used as parameter for contacts stream
- `sourceType` 'drift', required
- SourceDrip
- `api_key` string, required — API key to use. Find it at https://www.getdrip.com/user/edit
- `sourceType` 'drip', required
- SourceDropboxSign
- `api_key` string, required — API key to use. Find it at https://app.hellosign.com/home/myAccount#api
- `start_date` string, date-time, required
- `sourceType` 'dropbox-sign', required
- SourceDynamodb
- `credentials` object — Credentials for the service
- `auth_type` 'Role'
- `endpoint` string — the URL of the Dynamodb database
- `region` '' | 'af-south-1' | 'ap-east-1' | 'ap-northeast-1' | 'ap-northeast-2' | 'ap-northeast-3' | 'ap-south-1' | 'ap-south-2' | 'ap-southeast-1' | 'ap-southeast-2' | 'ap-southeast-3' | 'ap-southeast-4' | 'ca-central-1' | 'ca-west-1' | 'cn-north-1' | 'cn-northwest-1' | 'eu-central-1' | 'eu-central-2' | 'eu-north-1' | 'eu-south-1' | 'eu-south-2' | 'eu-west-1' | 'eu-west-2' | 'eu-west-3' | 'il-central-1' | 'me-central-1' | 'me-south-1' | 'sa-east-1' | 'us-east-1' | 'us-east-2' | 'us-gov-east-1' | 'us-gov-west-1' | 'us-west-1' | 'us-west-2' — The region of the Dynamodb database
- `reserved_attribute_names` string — Comma separated reserved attribute names present in your tables
- `ignore_missing_read_permissions_tables` boolean — Ignore tables with missing scan/read permissions
- `sourceType` 'dynamodb'
- SourceEasypost
- `username` string, required — The API Key from your easypost settings
- `start_date` string, date-time, required
- `sourceType` 'easypost', required
- SourceEmailoctopus
- `api_key` string, required — EmailOctopus API Key. See the docs for information on how to generate this key.
- `sourceType` 'emailoctopus', required
- SourceEmploymentHero
- `api_key` string, required
- `organization_configids` unknown[] — Organization ID which could be found as result of `organizations` stream to be used in other substreams
- unknown
- `employees_configids` unknown[] — Employees IDs in the given organisation found in `employees` stream for passing to sub-streams
- unknown
- `sourceType` 'employment-hero', required
- SourceEventbrite
- `private_token` string, required — The private token to use for authenticating API requests.
- `start_date` string, date-time, required
- `sourceType` 'eventbrite', required
- SourceExchangeRates
- `start_date` string, date, required — Start getting data from that date.
- `access_key` string, required — Your API Key. See here. The key is case sensitive.
- `base` string — ISO reference currency. See here. Free plan doesn't support Source Currency Switching, default base currency is EUR
- `ignore_weekends` boolean — Ignore weekends? (Exchanges don't run on weekends)
- `sourceType` 'exchange-rates', required
- SourceEzofficeinventory
- `api_key` string, required — Your EZOfficeInventory Access Token. API Access is disabled by default. Enable API Access in Settings > Integrations > API Integration and click on Update to generate a new access token
- `subdomain` string, required — The company name used in signup, also visible in the URL when logged in.
- `start_date` string, date-time, required — Earliest date you want to sync historical streams (inventory_histories, asset_histories, asset_stock_histories) from
- `sourceType` 'ezofficeinventory', required
- SourceFacebookMarketing
- `account_ids` string[], required — The Facebook Ad account ID(s) to pull data from. The Ad account ID number is in the account dropdown menu or in your browser's address bar of your Meta Ads Manager. See the docs for more information.
- `access_token` string — The value of the generated access token. From your App’s Dashboard, click on "Marketing API" then "Tools". Select permissions ads_management, ads_read, read_insights, business_management. Then click on "Get token". See the docs for more information.
- `credentials` union, required — Credentials for connecting to the Facebook Marketing API
- object
- `auth_type` 'Client', required
- `client_id` string, required — Client ID for the Facebook Marketing API
- `client_secret` string, required — Client Secret for the Facebook Marketing API
- `access_token` string — The value of the generated access token. From your App’s Dashboard, click on "Marketing API" then "Tools". Select permissions ads_management, ads_read, read_insights, business_management. Then click on "Get token". See the docs for more information.
- object
- `auth_type` 'Service', required
- `access_token` string, required — The value of the generated access token. From your App’s Dashboard, click on "Marketing API" then "Tools". Select permissions ads_management, ads_read, read_insights, business_management. Then click on "Get token". See the docs for more information.
- `start_date` string, date-time — The date from which you'd like to replicate data for all incremental streams, in the format YYYY-MM-DDT00:00:00Z. If not set then all data will be replicated for usual streams and only last 2 years for insight streams.
- `end_date` string, date-time — The date until which you'd like to replicate data for all incremental streams, in the format YYYY-MM-DDT00:00:00Z. All data generated between the start date and this end date will be replicated. Not setting this option will result in always syncing the latest data.
- `campaign_statuses` string[] — Select the statuses you want to be loaded in the stream. If no specific statuses are selected, the API's default behavior applies, and some statuses may be filtered out.
- `adset_statuses` string[] — Select the statuses you want to be loaded in the stream. If no specific statuses are selected, the API's default behavior applies, and some statuses may be filtered out.
- `ad_statuses` string[] — Select the statuses you want to be loaded in the stream. If no specific statuses are selected, the API's default behavior applies, and some statuses may be filtered out.
- `fetch_thumbnail_images` boolean — Set to active if you want to fetch the thumbnail_url and store the result in thumbnail_data_url for each Ad Creative.
- `custom_insights` object[] — A list which contains ad statistics entries, each entry must have a name and can contains fields, breakdowns or action_breakdowns. Click on "add" to fill this field.
- `name` string, required — The name value of insight
- `level` 'ad' | 'adset' | 'campaign' | 'account' — Chosen level for API
- `fields` string[] — A list of chosen fields for fields parameter
- `breakdowns` string[] — A list of chosen breakdowns for breakdowns
- `action_breakdowns` string[] — A list of chosen action_breakdowns for action_breakdowns
- `action_report_time` 'conversion' | 'impression' | 'mixed' — Determines the report time of action stats. For example, if a person saw the ad on Jan 1st but converted on Jan 2nd, when you query the API with action_report_time=impression, you see a conversion on Jan 1st. When you query the API with action_report_time=conversion, you see a conversion on Jan 2nd.
- `time_increment` integer — Time window in days by which to aggregate statistics. The sync will be chunked into N day intervals, where N is the number of days you specified. For example, if you set this value to 7, then all statistics will be reported as 7-day aggregates by starting from the start_date. If the start and end dates are October 1st and October 30th, then the connector will output 5 records: 01 - 06, 07 - 13, 14 - 20, 21 - 27, and 28 - 30 (3 days only). The minimum allowed value for this field is 1, and the maximum is 89.
- `start_date` string, date-time — The date from which you'd like to replicate data for this stream, in the format YYYY-MM-DDT00:00:00Z.
- `end_date` string, date-time — The date until which you'd like to replicate data for this stream, in the format YYYY-MM-DDT00:00:00Z. All data generated between the start date and this end date will be replicated. Not setting this option will result in always syncing the latest data.
- `insights_lookback_window` integer — The attribution window
- `insights_job_timeout` integer — The insights job timeout
- `page_size` integer — Page size used when sending requests to Facebook API to specify number of records per page when response has pagination. Most users do not need to set this field unless they specifically need to tune the connector to address specific issues or use cases.
- `insights_lookback_window` integer — The attribution window. Facebook freezes insight data 28 days after it was generated, which means that all data from the past 28 days may have changed since we last emitted it, so you can retrieve refreshed insights from the past by setting this parameter. If you set a custom lookback window value in Facebook account, please provide the same value here.
- `insights_job_timeout` integer — Insights Job Timeout establishes the maximum amount of time (in minutes) of waiting for the report job to complete. When timeout is reached the job is considered failed and we are trying to request smaller amount of data by breaking the job to few smaller ones. If you definitely know that 60 minutes is not enough for your report to be processed then you can decrease the timeout value, so we start breaking job to smaller parts faster.
- `sourceType` 'facebook-marketing', required
- SourceFactorial
- `api_key` string, required
- `start_date` string, date-time, required
- `limit` string — Max records per page limit
- `sourceType` 'factorial', required
- SourceFaker
- `count` integer — How many users should be generated in total. The purchases table will be scaled to match, with 10 purchases created per 10 users. This setting does not apply to the products stream.
- `seed` integer — Manually control the faker random seed to return the same values on subsequent runs (leave -1 for random)
- `records_per_slice` integer — How many fake records will be in each page (stream slice), before a state message is emitted?
- `always_updated` boolean — Should the updated_at values for every record be new each sync? Setting this to false will case the source to stop emitting records after COUNT records have been emitted.
- `parallelism` integer — How many parallel workers should we use to generate fake data? Choose a value equal to the number of CPUs you will allocate to this source.
- `sourceType` 'faker', required
- SourceFauna
- `domain` string, required — Domain of Fauna to query. Defaults db.fauna.com. See the docs.
- `port` integer, required — Endpoint port.
- `scheme` string, required — URL scheme.
- `secret` string, required — Fauna secret, used when authenticating with the database.
- `collection` object — Settings for the Fauna Collection.
- `page_size` integer, required — The page size used when reading documents from the database. The larger the page size, the faster the connector processes documents. However, if a page is too large, the connector may fail.
Choose your page size based on how large the documents are.
See the docs.
- `deletions` union, required — This only applies to incremental syncs.
Enabling deletion mode informs your destination of deleted documents.
Disabled - Leave this feature disabled, and ignore deleted documents.
Enabled - Enables this feature. When a document is deleted, the connector exports a record with a "deleted at" column containing the time that the document was deleted.
- object
- `deletion_mode` 'ignore', required
- object
- `deletion_mode` 'deleted_field', required
- `column` string, required — Name of the "deleted at" column.
- `sourceType` 'fauna', required
- SourceFile
- `dataset_name` string, required — The Name of the final table to replicate this file into (should include letters, numbers dash and underscores only).
- `format` 'csv' | 'json' | 'jsonl' | 'excel' | 'excel_binary' | 'fwf' | 'feather' | 'parquet' | 'yaml', required — The Format of the file which should be replicated (Warning: some formats may be experimental, please refer to the docs).
- `reader_options` string — This should be a string in JSON format. It depends on the chosen file format to provide additional options and tune its behavior.
- `url` string, required — The URL path to access the file which should be replicated.
- `provider` union, required — The storage Provider or Location of the file(s) which should be replicated.
- object
- `storage` 'HTTPS', required
- `user_agent` boolean — Add User-Agent to request
- object
- `storage` 'GCS', required
- `service_account_json` string — In order to access private Buckets stored on Google Cloud, this connector would need a service account json credentials with the proper permissions as described here. Please generate the credentials.json file and copy/paste its content to this field (expecting JSON formats). If accessing publicly available data, this field is not necessary.
- object
- `storage` 'S3', required
- `aws_access_key_id` string — In order to access private Buckets stored on AWS S3, this connector would need credentials with the proper permissions. If accessing publicly available data, this field is not necessary.
- `aws_secret_access_key` string — In order to access private Buckets stored on AWS S3, this connector would need credentials with the proper permissions. If accessing publicly available data, this field is not necessary.
- object
- `storage` 'AzBlob', required
- `storage_account` string, required — The globally unique name of the storage account that the desired blob sits within. See here for more details.
- `sas_token` string — To access Azure Blob Storage, this connector would need credentials with the proper permissions. One option is a SAS (Shared Access Signature) token. If accessing publicly available data, this field is not necessary.
- `shared_key` string — To access Azure Blob Storage, this connector would need credentials with the proper permissions. One option is a storage account shared key (aka account key or access key). If accessing publicly available data, this field is not necessary.
- object
- `storage` 'SSH', required
- `user` string, required
- `password` string
- `host` string, required
- `port` string
- object
- `storage` 'SCP', required
- `user` string, required
- `password` string
- `host` string, required
- `port` string
- object
- `storage` 'SFTP', required
- `user` string, required
- `password` string
- `host` string, required
- `port` string
- object
- `storage` 'local', required — WARNING: Note that the local storage URL available for reading must start with the local mount "/local/" at the moment until we implement more advanced docker mounting options.
- `sourceType` 'file', required
- SourceFirebolt
- `client_id` string, required — Firebolt service account ID.
- `client_secret` string, required — Firebolt secret, corresponding to the service account ID.
- `account` string, required — Firebolt account to login.
- `host` string — The host name of your Firebolt database.
- `database` string, required — The database to connect to.
- `engine` string, required — Engine name to connect to.
- `sourceType` 'firebolt', required
- SourceFleetio
- `api_key` string, required
- `account_token` string, required
- `sourceType` 'fleetio', required
- SourceFlexport
- `api_key` string, required
- `start_date` string, date-time, required
- `sourceType` 'flexport', required
- SourceFreeAgentConnector
- `client_id` string, required
- `client_secret` string, required
- `client_refresh_token_2` string, required
- `updated_since` string, date-time
- `payroll_year` number
- `sourceType` 'free-agent-connector', required
- SourceFreshcaller
- `domain` string, required — Used to construct Base URL for the Freshcaller APIs
- `api_key` string, required — Freshcaller API Key. See the docs for more information on how to obtain this key.
- `requests_per_minute` integer — The number of requests per minute that this source allowed to use. There is a rate limit of 50 requests per minute per app per account.
- `start_date` string, date-time — UTC date and time. Any data created after this date will be replicated.
- `sync_lag_minutes` integer — Lag in minutes for each sync, i.e., at time T, data for the time range [prev_sync_time, T-30] will be fetched
- `sourceType` 'freshcaller', required
- SourceFreshchat
- `api_key` string, required
- `start_date` string, date-time, required
- `account_name` string, required — The unique account name for your Freshchat instance
- `sourceType` 'freshchat', required
- SourceFreshdesk
- `api_key` string, required — Freshdesk API Key. See the docs for more information on how to obtain this key.
- `domain` string, required — Freshdesk domain
- `requests_per_minute` integer — The number of requests per minute that this source allowed to use. There is a rate limit of 50 requests per minute per app per account.
- `start_date` string, date-time — UTC date and time. Any data created after this date will be replicated. If this parameter is not set, all data will be replicated.
- `lookback_window_in_days` integer — Number of days for lookback window for the stream Satisfaction Ratings
- `sourceType` 'freshdesk', required
- SourceFreshsales
- `api_key` string, required — Freshsales API Key. See here. The key is case sensitive.
- `domain_name` string, required — The Name of your Freshsales domain
- `sourceType` 'freshsales', required
- SourceFreshservice
- `api_key` string, required — Freshservice API Key. See here. The key is case sensitive.
- `domain_name` string, required — The name of your Freshservice domain
- `start_date` string, date-time, required — UTC date and time in the format 2020-10-01T00:00:00Z. Any data before this date will not be replicated.
- `sourceType` 'freshservice', required
- SourceFront
- `api_key` string, required
- `start_date` string, date-time, required
- `page_limit` string — Page limit for the responses
- `sourceType` 'front', required
- SourceGainsightPx
- `api_key` string, required — The Aptrinsic API Key which is recieved from the dashboard settings (ref - https://app.aptrinsic.com/settings/api-keys)
- `sourceType` 'gainsight-px', required
- SourceGcs — NOTE: When this Spec is changed, legacy_config_transformer.py must also be modified to uptake the changes because it is responsible for converting legacy GCS configs into file based configs using the File-Based CDK.
- `start_date` string, date-time — UTC date and time in the format 2017-01-25T00:00:00.000000Z. Any file modified before this date will not be replicated.
- `streams` object[], required — Each instance of this configuration defines a stream. Use this to define which files belong in the stream, their format, and how they should be parsed and validated. When sending data to warehouse destination such as Snowflake or BigQuery, each stream is a separate table.
- `name` string, required — The name of the stream.
- `globs` string[] — The pattern used to specify which files should be selected from the file system. For more information on glob pattern matching look here.
- `validation_policy` 'Emit Record' | 'Skip Record' | 'Wait for Discover' — The name of the validation policy that dictates sync behavior when a record does not adhere to the stream schema.
- `input_schema` string — The schema that will be used to validate records extracted from the file. This will override the stream schema that is auto-detected from incoming files.
- `days_to_sync_if_history_is_full` integer — When the state history of the file store is full, syncs will only read files that were last modified in the provided day range.
- `format` union, required — The configuration options that are used to alter how to read incoming files that deviate from the standard formatting.
- object
- `filetype` 'avro', required
- `double_as_string` boolean — Whether to convert double fields to strings. This is recommended if you have decimal numbers with a high degree of precision because there can be a loss precision when handling floating point numbers.
- object
- `filetype` 'csv', required
- `delimiter` string — The character delimiting individual cells in the CSV data. This may only be a 1-character string. For tab-delimited data enter '\t'.
- `quote_char` string — The character used for quoting CSV values. To disallow quoting, make this field blank.
- `escape_char` string — The character used for escaping special characters. To disallow escaping, leave this field blank.
- `encoding` string — The character encoding of the CSV data. Leave blank to default to UTF8. See list of python encodings for allowable options.
- `double_quote` boolean — Whether two quotes in a quoted CSV value denote a single quote in the data.
- `null_values` string[] — A set of case-sensitive strings that should be interpreted as null values. For example, if the value 'NA' should be interpreted as null, enter 'NA' in this field.
- `strings_can_be_null` boolean — Whether strings can be interpreted as null values. If true, strings that match the null_values set will be interpreted as null. If false, strings that match the null_values set will be interpreted as the string itself.
- `skip_rows_before_header` integer — The number of rows to skip before the header row. For example, if the header row is on the 3rd row, enter 2 in this field.
- `skip_rows_after_header` integer — The number of rows to skip after the header row.
- `header_definition` union — How headers will be defined. `User Provided` assumes the CSV does not have a header row and uses the headers provided and `Autogenerated` assumes the CSV does not have a header row and the CDK will generate headers using for `f{i}` where `i` is the index starting from 0. Else, the default behavior is to use the header from the CSV file. If a user wants to autogenerate or provide column names for a CSV having headers, they can skip rows.
- …
- `true_values` string[] — A set of case-sensitive strings that should be interpreted as true values.
- `false_values` string[] — A set of case-sensitive strings that should be interpreted as false values.
- `ignore_errors_on_fields_mismatch` boolean — Whether to ignore errors that occur when the number of fields in the CSV does not match the number of columns in the schema.
- object
- `filetype` 'jsonl', required
- object
- `filetype` 'parquet', required
- `decimal_as_float` boolean — Whether to convert decimal fields to floats. There is a loss of precision when converting decimals to floats, so this is not recommended.
- object — Extract text from document formats (.pdf, .docx, .md, .pptx) and emit as one record per file.
- `filetype` 'unstructured', required
- `skip_unprocessable_files` boolean — If true, skip files that cannot be parsed and pass the error message along as the _ab_source_file_parse_error field. If false, fail the sync.
- `strategy` 'auto' | 'fast' | 'ocr_only' | 'hi_res' — The strategy used to parse documents. `fast` extracts text directly from the document which doesn't work for all files. `ocr_only` is more reliable, but slower. `hi_res` is the most reliable, but requires an API key and a hosted instance of unstructured and can't be used with local mode. See the unstructured.io documentation for more details: https://unstructured-io.github.io/unstructured/core/partition.html#partition-pdf
- `processing` union — Processing configuration
- …
- object
- `filetype` 'excel', required
- `schemaless` boolean — When enabled, syncs will not validate or structure records against the stream's schema.
- `recent_n_files_to_read_for_schema_discovery` integer — The number of resent files which will be used to discover the schema for this stream.
- `service_account` string, required — Enter your Google Cloud service account key in JSON format
- `bucket` string, required — Name of the GCS bucket where the file(s) exist.
- `sourceType` 'gcs', required
- SourceGetlago
- `api_url` string — Your Lago API URL
- `api_key` string, required — Your API Key. See here.
- `sourceType` 'getlago', required
- SourceGithub
- `credentials` union, required — Choose how to authenticate to GitHub
- object
- `option_title` 'OAuth Credentials'
- `access_token` string, required — OAuth access token
- `client_id` string — OAuth Client Id
- `client_secret` string — OAuth Client secret
- object
- `option_title` 'PAT Credentials'
- `personal_access_token` string, required — Log into GitHub and then generate a personal access token. To load balance your API quota consumption across multiple API tokens, input multiple tokens separated with ","
- `repositories` string[], required — List of GitHub organizations/repositories, e.g. `airbytehq/airbyte` for single repository, `airbytehq/*` for get all repositories from organization and `airbytehq/a* for matching multiple repositories by pattern.
- `start_date` string, date-time — The date from which you'd like to replicate data from GitHub in the format YYYY-MM-DDT00:00:00Z. If the date is not set, all data will be replicated. For the streams which support this configuration, only data generated on or after the start date will be replicated. This field doesn't apply to all streams, see the docs for more info
- `api_url` string — Please enter your basic URL from self-hosted GitHub instance or leave it empty to use GitHub.
- `branches` string[] — List of GitHub repository branches to pull commits for, e.g. `airbytehq/airbyte/master`. If no branches are specified for a repository, the default branch will be pulled.
- `max_waiting_time` integer — Max Waiting Time for rate limit. Set higher value to wait till rate limits will be resetted to continue sync
- `sourceType` 'github', required
- SourceGitlab
- `credentials` union, required
- object
- `auth_type` 'oauth2.0'
- `client_id` string, required — The API ID of the Gitlab developer application.
- `client_secret` string, required — The API Secret the Gitlab developer application.
- `access_token` string, required — Access Token for making authenticated requests.
- `token_expiry_date` string, date-time, required — The date-time when the access token should be refreshed.
- `refresh_token` string, required — The key to refresh the expired access_token.
- object
- `auth_type` 'access_token'
- `access_token` string, required — Log into your Gitlab account and then generate a personal Access Token.
- `start_date` string, date-time — The date from which you'd like to replicate data for GitLab API, in the format YYYY-MM-DDT00:00:00Z. Optional. If not set, all data will be replicated. All data generated after this date will be replicated.
- `api_url` string — Please enter your basic URL from GitLab instance.
- `groups_list` string[] — List of groups. e.g. airbyte.io.
- `projects_list` string[] — Space-delimited list of projects. e.g. airbyte.io/documentation meltano/tap-gitlab.
- `sourceType` 'gitlab', required
- SourceGlassfrog
- `api_key` string, required — API key provided by Glassfrog
- `sourceType` 'glassfrog', required
- SourceGmail
- `client_id` string, required
- `client_secret` string, required
- `client_refresh_token` string, required
- `include_spam_and_trash` boolean — Include drafts/messages from SPAM and TRASH in the results. Defaults to false.
- `sourceType` 'gmail', required
- SourceGnews
- `api_key` string, required — API Key
- `query` string, required — This parameter allows you to specify your search keywords to find the news articles you are looking for. The keywords will be used to return the most relevant articles. It is possible to use logical operators with keywords. - Phrase Search Operator: This operator allows you to make an exact search. Keywords surrounded by quotation marks are used to search for articles with the exact same keyword sequence. For example the query: "Apple iPhone" will return articles matching at least once this sequence of keywords. - Logical AND Operator: This operator allows you to make sure that several keywords are all used in the article search. By default the space character acts as an AND operator, it is possible to replace the space character by AND to obtain the same result. For example the query: Apple Microsoft is equivalent to Apple AND Microsoft - Logical OR Operator: This operator allows you to retrieve articles matching the keyword a or the keyword b. It is important to note that this operator has a higher precedence than the AND operator. For example the query: Apple OR Microsoft will return all articles matching the keyword Apple as well as all articles matching the keyword Microsoft - Logical NOT Operator: This operator allows you to remove from the results the articles corresponding to the specified keywords. To use it, you need to add NOT in front of each word or phrase surrounded by quotes. For example the query: Apple NOT iPhone will return all articles matching the keyword Apple but not the keyword iPhone
- `language` 'ar' | 'zh' | 'nl' | 'en' | 'fr' | 'de' | 'el' | 'he' | 'hi' | 'it' | 'ja' | 'ml' | 'mr' | 'no' | 'pt' | 'ro' | 'ru' | 'es' | 'sv' | 'ta' | 'te' | 'uk'
- `country` 'au' | 'br' | 'ca' | 'cn' | 'eg' | 'fr' | 'de' | 'gr' | 'hk' | 'in' | 'ie' | 'il' | 'it' | 'jp' | 'nl' | 'no' | 'pk' | 'pe' | 'ph' | 'pt' | 'ro' | 'ru' | 'sg' | 'es' | 'se' | 'ch' | 'tw' | 'ua' | 'gb' | 'us' — This parameter allows you to specify the country where the news articles returned by the API were published, the contents of the articles are not necessarily related to the specified country. You have to set as value the 2 letters code of the country you want to filter.
- `in` string[] — This parameter allows you to choose in which attributes the keywords are searched. The attributes that can be set are title, description and content. It is possible to combine several attributes.
- `nullable` string[] — This parameter allows you to specify the attributes that you allow to return null values. The attributes that can be set are title, description and content. It is possible to combine several attributes
- `start_date` string — This parameter allows you to filter the articles that have a publication date greater than or equal to the specified value. The date must respect the following format: YYYY-MM-DD hh:mm:ss (in UTC)
- `end_date` string — This parameter allows you to filter the articles that have a publication date smaller than or equal to the specified value. The date must respect the following format: YYYY-MM-DD hh:mm:ss (in UTC)
- `sortby` 'publishedAt' | 'relevance' — This parameter allows you to choose with which type of sorting the articles should be returned. Two values are possible: - publishedAt = sort by publication date, the articles with the most recent publication date are returned first - relevance = sort by best match to keywords, the articles with the best match are returned first
- `top_headlines_query` string — This parameter allows you to specify your search keywords to find the news articles you are looking for. The keywords will be used to return the most relevant articles. It is possible to use logical operators with keywords. - Phrase Search Operator: This operator allows you to make an exact search. Keywords surrounded by quotation marks are used to search for articles with the exact same keyword sequence. For example the query: "Apple iPhone" will return articles matching at least once this sequence of keywords. - Logical AND Operator: This operator allows you to make sure that several keywords are all used in the article search. By default the space character acts as an AND operator, it is possible to replace the space character by AND to obtain the same result. For example the query: Apple Microsoft is equivalent to Apple AND Microsoft - Logical OR Operator: This operator allows you to retrieve articles matching the keyword a or the keyword b. It is important to note that this operator has a higher precedence than the AND operator. For example the query: Apple OR Microsoft will return all articles matching the keyword Apple as well as all articles matching the keyword Microsoft - Logical NOT Operator: This operator allows you to remove from the results the articles corresponding to the specified keywords. To use it, you need to add NOT in front of each word or phrase surrounded by quotes. For example the query: Apple NOT iPhone will return all articles matching the keyword Apple but not the keyword iPhone
- `top_headlines_topic` 'breaking-news' | 'world' | 'nation' | 'business' | 'technology' | 'entertainment' | 'sports' | 'science' | 'health' — This parameter allows you to change the category for the request.
- `sourceType` 'gnews', required
- SourceGocardless
- `access_token` string, required — Gocardless API TOKEN
- `gocardless_environment` 'sandbox' | 'live', required — Environment you are trying to connect to.
- `gocardless_version` string, required — GoCardless version. This is a date. You can find the latest here: https://developer.gocardless.com/api-reference/#api-usage-making-requests
- `start_date` string, required — UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.
- `sourceType` 'gocardless', required
- SourceGoldcast
- `access_key` string, required — Your API Access Key. See here. The key is case sensitive.
- `sourceType` 'goldcast', required
- SourceGong
- `access_key` string, required — Gong Access Key
- `access_key_secret` string, required — Gong Access Key Secret
- `start_date` string — The date from which to list calls, in the ISO-8601 format; if not specified, the calls start with the earliest recorded call. For web-conference calls recorded by Gong, the date denotes its scheduled time, otherwise, it denotes its actual start time.
- `sourceType` 'gong', required
- SourceGoogleAds
- `credentials` object, required
- `developer_token` string, required — The Developer Token granted by Google to use their APIs. For detailed instructions on finding this value, refer to our documentation.
- `client_id` string, required — The Client ID of your Google Ads developer application. For detailed instructions on finding this value, refer to our documentation.
- `client_secret` string, required — The Client Secret of your Google Ads developer application. For detailed instructions on finding this value, refer to our documentation.
- `refresh_token` string, required — The token used to obtain a new Access Token. For detailed instructions on finding this value, refer to our documentation.
- `access_token` string — The Access Token for making authenticated requests. For detailed instructions on finding this value, refer to our documentation.
- `customer_id` string — Comma-separated list of (client) customer IDs. Each customer ID must be specified as a 10-digit number without dashes. For detailed instructions on finding this value, refer to our documentation.
- `customer_status_filter` string[] — A list of customer statuses to filter on. For detailed info about what each status mean refer to Google Ads documentation.
- `start_date` string, date — UTC date in the format YYYY-MM-DD. Any data before this date will not be replicated. (Default value of two years ago is used if not set)
- `end_date` string, date — UTC date in the format YYYY-MM-DD. Any data after this date will not be replicated. (Default value of today is used if not set)
- `custom_queries_array` object[]
- `query` string, required — A custom defined GAQL query for building the report. Avoid including the segments.date field; wherever possible, Airbyte will automatically include it for incremental syncs. For more information, refer to Google's documentation.
- `table_name` string, required — The table name in your destination database for the chosen query.
- `conversion_window_days` integer — A conversion window is the number of days after an ad interaction (such as an ad click or video view) during which a conversion, such as a purchase, is recorded in Google Ads. For more information, see Google's documentation.
- `sourceType` 'google-ads', required
- SourceGoogleAnalyticsDataApi
- `credentials` union — Credentials for the service
- object
- `auth_type` 'Client'
- `client_id` string, required — The Client ID of your Google Analytics developer application.
- `client_secret` string, required — The Client Secret of your Google Analytics developer application.
- `refresh_token` string, required — The token for obtaining a new access token.
- `access_token` string — Access Token for making authenticated requests.
- object
- `auth_type` 'Service'
- `credentials_json` string, required — The JSON key linked to the service account used for authorization. For steps on obtaining this key, refer to the setup guide.
- `property_ids` string[], required — A list of your Property IDs. The Property ID is a unique number assigned to each property in Google Analytics, found in your GA4 property URL. This ID allows the connector to track the specific events associated with your property. Refer to the Google Analytics documentation to locate your property ID.
- `date_ranges_start_date` string, date — The start date from which to replicate report data in the format YYYY-MM-DD. Data generated before this date will not be included in the report. Not applied to custom Cohort reports.
- `custom_reports_array` object[] — You can add your Custom Analytics report by creating one.
- `name` string, required — The name of the custom report, this name would be used as stream name.
- `dimensions` string[], required — A list of dimensions.
- `metrics` string[], required — A list of metrics.
- `dimensionFilter` union — Dimensions filter
- object — The FilterExpressions in andGroup have an AND relationship.
- `filter_type` 'andGroup', required
- `expressions` object[], required
- …
- object — The FilterExpressions in orGroup have an OR relationship.
- `filter_type` 'orGroup', required
- `expressions` object[], required
- …
- object — The FilterExpression is NOT of notExpression.
- `filter_type` 'notExpression'
- `expression` object
- …
- object — A primitive filter. In the same FilterExpression, all of the filter's field names need to be either all dimensions.
- `filter_type` 'filter'
- `field_name` string, required
- `filter` union, required
- …
- `metricFilter` union — Metrics filter
- object — The FilterExpressions in andGroup have an AND relationship.
- `filter_type` 'andGroup', required
- `expressions` object[], required
- …
- object — The FilterExpressions in orGroup have an OR relationship.
- `filter_type` 'orGroup', required
- `expressions` object[], required
- …
- object — The FilterExpression is NOT of notExpression.
- `filter_type` 'notExpression'
- `expression` object
- …
- object — A primitive filter. In the same FilterExpression, all of the filter's field names need to be either all metrics.
- `filter_type` 'filter'
- `field_name` string, required
- `filter` union, required
- …
- `cohortSpec` union — Cohort reports creates a time series of user retention for the cohort.
- object
- `enabled` 'false'
- object
- `enabled` 'true'
- `cohorts` object[]
- …
- `cohortsRange` object
- …
- `cohortReportSettings` object — Optional settings for a cohort report.
- …
- `window_in_days` integer — The interval in days for each data request made to the Google Analytics API. A larger value speeds up data sync, but increases the chance of data sampling, which may result in inaccuracies. We recommend a value of 1 to minimize sampling, unless speed is an absolute priority over accuracy. Acceptable values range from 1 to 364. Does not apply to custom Cohort reports. More information is available in the documentation.
- `lookback_window` integer — Since attribution changes after the event date, and Google Analytics has a data processing latency, we should specify how many days in the past we should refresh the data in every run. So if you set it at 5 days, in every sync it will fetch the last bookmark date minus 5 days.
- `keep_empty_rows` boolean — If false, each row with all metrics equal to 0 will not be returned. If true, these rows will be returned if they are not separately removed by a filter. More information is available in the documentation.
- `convert_conversions_event` boolean — Enables conversion of `conversions:*` event metrics from integers to floats. This is beneficial for preventing data rounding when the API returns float values for any `conversions:*` fields.
- `sourceType` 'google-analytics-data-api', required
- SourceGoogleCalendar
- `client_id` string, required
- `client_secret` string, required
- `client_refresh_token_2` string, required
- `calendarid` string, required
- `sourceType` 'google-calendar', required
- SourceGoogleDirectory
- `credentials` union — Google APIs use the OAuth 2.0 protocol for authentication and authorization. The Source supports Web server application and Service accounts scenarios.
- object — For these scenario user only needs to give permission to read Google Directory data.
- `credentials_title` 'Web server app' — Authentication Scenario
- `client_id` string, required — The Client ID of the developer application.
- `client_secret` string, required — The Client Secret of the developer application.
- `refresh_token` string, required — The Token for obtaining a new access token.
- object — For these scenario user should obtain service account's credentials from the Google API Console and provide delegated email.
- `credentials_title` 'Service accounts' — Authentication Scenario
- `credentials_json` string, required — The contents of the JSON service account key. See the docs for more information on how to generate this key.
- `email` string, required — The email of the user, which has permissions to access the Google Workspace Admin APIs.
- `sourceType` 'google-directory', required
- SourceGoogleDrive — Used during spec; allows the developer to configure the cloud provider specific options that are needed when users configure a file-based source.
- `start_date` string, date-time — UTC date and time in the format 2017-01-25T00:00:00.000000Z. Any file modified before this date will not be replicated.
- `streams` object[], required — Each instance of this configuration defines a stream. Use this to define which files belong in the stream, their format, and how they should be parsed and validated. When sending data to warehouse destination such as Snowflake or BigQuery, each stream is a separate table.
- `name` string, required — The name of the stream.
- `globs` string[] — The pattern used to specify which files should be selected from the file system. For more information on glob pattern matching look here.
- `validation_policy` 'Emit Record' | 'Skip Record' | 'Wait for Discover' — The name of the validation policy that dictates sync behavior when a record does not adhere to the stream schema.
- `input_schema` string — The schema that will be used to validate records extracted from the file. This will override the stream schema that is auto-detected from incoming files.
- `days_to_sync_if_history_is_full` integer — When the state history of the file store is full, syncs will only read files that were last modified in the provided day range.
- `format` union, required — The configuration options that are used to alter how to read incoming files that deviate from the standard formatting.
- object
- `filetype` 'avro', required
- `double_as_string` boolean — Whether to convert double fields to strings. This is recommended if you have decimal numbers with a high degree of precision because there can be a loss precision when handling floating point numbers.
- object
- `filetype` 'csv', required
- `delimiter` string — The character delimiting individual cells in the CSV data. This may only be a 1-character string. For tab-delimited data enter '\t'.
- `quote_char` string — The character used for quoting CSV values. To disallow quoting, make this field blank.
- `escape_char` string — The character used for escaping special characters. To disallow escaping, leave this field blank.
- `encoding` string — The character encoding of the CSV data. Leave blank to default to UTF8. See list of python encodings for allowable options.
- `double_quote` boolean — Whether two quotes in a quoted CSV value denote a single quote in the data.
- `null_values` string[] — A set of case-sensitive strings that should be interpreted as null values. For example, if the value 'NA' should be interpreted as null, enter 'NA' in this field.
- `strings_can_be_null` boolean — Whether strings can be interpreted as null values. If true, strings that match the null_values set will be interpreted as null. If false, strings that match the null_values set will be interpreted as the string itself.
- `skip_rows_before_header` integer — The number of rows to skip before the header row. For example, if the header row is on the 3rd row, enter 2 in this field.
- `skip_rows_after_header` integer — The number of rows to skip after the header row.
- `header_definition` union — How headers will be defined. `User Provided` assumes the CSV does not have a header row and uses the headers provided and `Autogenerated` assumes the CSV does not have a header row and the CDK will generate headers using for `f{i}` where `i` is the index starting from 0. Else, the default behavior is to use the header from the CSV file. If a user wants to autogenerate or provide column names for a CSV having headers, they can skip rows.
- …
- `true_values` string[] — A set of case-sensitive strings that should be interpreted as true values.
- `false_values` string[] — A set of case-sensitive strings that should be interpreted as false values.
- `ignore_errors_on_fields_mismatch` boolean — Whether to ignore errors that occur when the number of fields in the CSV does not match the number of columns in the schema.
- object
- `filetype` 'jsonl', required
- object
- `filetype` 'parquet', required
- `decimal_as_float` boolean — Whether to convert decimal fields to floats. There is a loss of precision when converting decimals to floats, so this is not recommended.
- object — Extract text from document formats (.pdf, .docx, .md, .pptx) and emit as one record per file.
- `filetype` 'unstructured', required
- `skip_unprocessable_files` boolean — If true, skip files that cannot be parsed and pass the error message along as the _ab_source_file_parse_error field. If false, fail the sync.
- `strategy` 'auto' | 'fast' | 'ocr_only' | 'hi_res' — The strategy used to parse documents. `fast` extracts text directly from the document which doesn't work for all files. `ocr_only` is more reliable, but slower. `hi_res` is the most reliable, but requires an API key and a hosted instance of unstructured and can't be used with local mode. See the unstructured.io documentation for more details: https://unstructured-io.github.io/unstructured/core/partition.html#partition-pdf
- `processing` object — Process files locally, supporting `fast` and `ocr` modes. This is the default option.
- …
- `schemaless` boolean — When enabled, syncs will not validate or structure records against the stream's schema.
- `folder_url` string, required — URL for the folder you want to sync. Using individual streams and glob patterns, it's possible to only sync a subset of all files located in the folder.
- `credentials` union, required — Credentials for connecting to the Google Drive API
- object
- `auth_type` 'Client', required
- `client_id` string, required — Client ID for the Google Drive API
- `client_secret` string, required — Client Secret for the Google Drive API
- `refresh_token` string, required — Refresh Token for the Google Drive API
- object
- `auth_type` 'Service', required
- `service_account_info` string, required — The JSON key of the service account to use for authorization. Read more here.
- `sourceType` 'google-drive', required
- SourceGooglePagespeedInsights
- `api_key` string — Google PageSpeed API Key. See here. The key is optional - however the API is heavily rate limited when using without API Key. Creating and using the API key therefore is recommended. The key is case sensitive.
- `categories` string[], required — Defines which Lighthouse category to run. One or many of: "accessibility", "best-practices", "performance", "pwa", "seo".
- `strategies` string[], required — The analyses strategy to use. Either "desktop" or "mobile".
- `urls` string[], required — The URLs to retrieve pagespeed information from. The connector will attempt to sync PageSpeed reports for all the defined URLs. Format: https://(www.)url.domain
- `sourceType` 'google-pagespeed-insights', required
- SourceGoogleSearchConsole
- `site_urls` string[], required — The URLs of the website property attached to your GSC account. Learn more about properties here.
- `start_date` string, date — UTC date in the format YYYY-MM-DD. Any data before this date will not be replicated.
- `end_date` string, date — UTC date in the format YYYY-MM-DD. Any data created after this date will not be replicated. Must be greater or equal to the start date field. Leaving this field blank will replicate all data from the start date onward.
- `authorization` union, required
- object
- `auth_type` 'Client', required
- `client_id` string, required — The client ID of your Google Search Console developer application. Read more here.
- `client_secret` string, required — The client secret of your Google Search Console developer application. Read more here.
- `access_token` string — Access token for making authenticated requests. Read more here.
- `refresh_token` string, required — The token for obtaining a new access token. Read more here.
- object
- `auth_type` 'Service', required
- `service_account_info` string, required — The JSON key of the service account to use for authorization. Read more here.
- `email` string, required — The email of the user which has permissions to access the Google Workspace Admin APIs.
- `custom_reports_array` object[] — You can add your Custom Analytics report by creating one.
- `name` string, required — The name of the custom report, this name would be used as stream name
- `dimensions` string[], required — A list of available dimensions. Please note, that for technical reasons `date` is the default dimension which will be included in your query whether you specify it or not. Primary key will consist of your custom dimensions and the default dimension along with `site_url` and `search_type`.
- `data_state` 'final' | 'all' — If set to 'final', the returned data will include only finalized, stable data. If set to 'all', fresh data will be included. When using Incremental sync mode, we do not recommend setting this parameter to 'all' as it may cause data loss. More information can be found in our full documentation.
- `sourceType` 'google-search-console', required
- SourceGoogleSheets
- `batch_size` integer — Default value is 200. An integer representing row batch size for each sent request to Google Sheets API. Row batch size means how many rows are processed from the google sheet, for example default value 200 would process rows 1-201, then 201-401 and so on. Based on Google Sheets API limits documentation, it is possible to send up to 300 requests per minute, but each individual request has to be processed under 180 seconds, otherwise the request returns a timeout error. In regards to this information, consider network speed and number of columns of the google sheet when deciding a batch_size value. Default value should cover most of the cases, but if a google sheet has over 100,000 records or more, consider increasing batch_size value.
- `spreadsheet_id` string, required — Enter the link to the Google spreadsheet you want to sync. To copy the link, click the 'Share' button in the top-right corner of the spreadsheet, then click 'Copy link'.
- `names_conversion` boolean — Enables the conversion of column names to a standardized, SQL-compliant format. For example, 'My Name' -> 'my_name'. Enable this option if your destination is SQL-based.
- `credentials` union, required — Credentials for connecting to the Google Sheets API
- object
- `auth_type` 'Client', required
- `client_id` string, required — Enter your Google application's Client ID. See Google's documentation for more information.
- `client_secret` string, required — Enter your Google application's Client Secret. See Google's documentation for more information.
- `refresh_token` string, required — Enter your Google application's refresh token. See Google's documentation for more information.
- object
- `auth_type` 'Service', required
- `service_account_info` string, required — The JSON key of the service account to use for authorization. Read more here.
- `sourceType` 'google-sheets', required
- SourceGoogleTasks
- `api_key` string, required
- `start_date` string, date-time, required
- `records_limit` string — The maximum number of records to be returned per request
- `sourceType` 'google-tasks', required
- SourceGoogleWebfonts
- `api_key` string, required — API key is required to access google apis, For getting your's goto google console and generate api key for Webfonts
- `alt` string — Optional, Available params- json, media, proto
- `prettyPrint` string — Optional, boolean type
- `sort` string — Optional, to find how to sort
- `sourceType` 'google-webfonts', required
- SourceGorgias
- `username` string, required
- `password` string
- `domain_name` string, required — Domain name given for gorgias, found as your url prefix for accessing your website
- `start_date` string, date-time, required
- `sourceType` 'gorgias', required
- SourceGreenhouse
- `api_key` string, required — Greenhouse API Key. See the docs for more information on how to generate this key.
- `sourceType` 'greenhouse', required
- SourceGridly
- `api_key` string, required
- `grid_id` string, required — ID of a grid, or can be ID of a branch
- `sourceType` 'gridly', required
- SourceGuru
- `username` string, required
- `password` string
- `start_date` string, date-time, required
- `team_id` string — Team ID received through response of /teams streams, make sure about access to the team
- `search_cards_query` string — Query for searching cards
- `sourceType` 'guru', required
- SourceGutendex
- `author_year_end` string — (Optional) Defines the maximum birth year of the authors. Books by authors born after the end year will not be returned. Supports both positive (CE) or negative (BCE) integer values
- `author_year_start` string — (Optional) Defines the minimum birth year of the authors. Books by authors born prior to the start year will not be returned. Supports both positive (CE) or negative (BCE) integer values
- `copyright` string — (Optional) Use this to find books with a certain copyright status - true for books with existing copyrights, false for books in the public domain in the USA, or null for books with no available copyright information.
- `languages` string — (Optional) Use this to find books in any of a list of languages. They must be comma-separated, two-character language codes.
- `search` string — (Optional) Use this to search author names and book titles with given words. They must be separated by a space (i.e. %20 in URL-encoded format) and are case-insensitive.
- `sort` string — (Optional) Use this to sort books - ascending for Project Gutenberg ID numbers from lowest to highest, descending for IDs highest to lowest, or popular (the default) for most popular to least popular by number of downloads.
- `topic` string — (Optional) Use this to search for a case-insensitive key-phrase in books' bookshelves or subjects.
- `sourceType` 'gutendex', required
- SourceHardcodedRecords
- `count` integer — How many records per stream should be generated
- `sourceType` 'hardcoded-records', required
- SourceHarvest
- `account_id` string, required — Harvest account ID. Required for all Harvest requests in pair with Personal Access Token
- `replication_start_date` string, date-time, required — UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.
- `credentials` union — Choose how to authenticate to Harvest.
- object
- `auth_type` 'Client'
- `client_id` string, required — The Client ID of your Harvest developer application.
- `client_secret` string, required — The Client Secret of your Harvest developer application.
- `refresh_token` string, required — Refresh Token to renew the expired Access Token.
- object
- `api_token` string, required — Log into Harvest and then create new personal access token.
- `auth_type` 'Token'
- `sourceType` 'harvest', required
- SourceHeight
- `api_key` string, required
- `start_date` string, date-time, required
- `search_query` string — Search query to be used with search stream
- `sourceType` 'height', required
- SourceHibob
- `username` string, required
- `password` string
- `is_sandbox` boolean, required — Toggle true if this instance is a HiBob sandbox
- `sourceType` 'hibob', required
- SourceHighLevel
- `location_id` string, required
- `api_key` string, required
- `start_date` string, date-time, required
- `sourceType` 'high-level', required
- SourceHubplanner
- `api_key` string, required — Hubplanner API key. See https://github.com/hubplanner/API#authentication for more details.
- `sourceType` 'hubplanner', required
- SourceHubspot
- `start_date` string, date-time — UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated. If not set, "2006-06-01T00:00:00Z" (Hubspot creation date) will be used as start date. It's recommended to provide relevant to your data start date value to optimize synchronization.
- `credentials` union, required — Choose how to authenticate to HubSpot.
- object
- `credentials_title` 'OAuth Credentials', required — Name of the credentials
- `client_id` string, required — The Client ID of your HubSpot developer application. See the Hubspot docs if you need help finding this ID.
- `client_secret` string, required — The client secret for your HubSpot developer application. See the Hubspot docs if you need help finding this secret.
- `refresh_token` string, required — Refresh token to renew an expired access token. See the Hubspot docs if you need help finding this token.
- object
- `credentials_title` 'Private App Credentials', required — Name of the credentials set
- `access_token` string, required — HubSpot Access token. See the Hubspot docs if you need help finding this token.
- `enable_experimental_streams` boolean — If enabled then experimental streams become available for sync.
- `sourceType` 'hubspot', required
- SourceIlluminaBasespace
- `access_token` string, required — BaseSpace access token. Instructions for obtaining your access token can be found in the BaseSpace Developer Documentation.
- `domain` string, required — Domain name of the BaseSpace instance (e.g., euw2.sh.basespace.illumina.com)
- `user` string, required — Providing a user ID restricts the returned data to what that user can access. If you use the default ('current'), all data accessible to the user associated with the API key will be shown.
- `sourceType` 'illumina-basespace', required
- SourceIncidentIo
- `api_key` string, required — API key to use. Find it at https://app.incident.io/settings/api-keys
- `sourceType` 'incident-io', required
- SourceInsightly
- `start_date` string, date-time, nullable, required — The date from which you'd like to replicate data for Insightly in the format YYYY-MM-DDT00:00:00Z. All data generated after this date will be replicated. Note that it will be used only for incremental streams.
- `token` string, nullable, required — Your Insightly API token.
- `sourceType` 'insightly', required
- SourceInstagram
- `start_date` string, date-time — The date from which you'd like to replicate data for User Insights, in the format YYYY-MM-DDT00:00:00Z. All data generated after this date will be replicated. If left blank, the start date will be set to 2 years before the present date.
- `access_token` string, required — The value of the access token generated with instagram_basic, instagram_manage_insights, pages_show_list, pages_read_engagement, Instagram Public Content Access permissions. See the docs for more information
- `sourceType` 'instagram', required
- SourceInstatus
- `api_key` string, required — Instatus REST API key
- `sourceType` 'instatus', required
- SourceIntercom
- `start_date` string, date-time, required — UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.
- `access_token` string, required — Access token for making authenticated requests. See the Intercom docs for more information.
- `client_id` string — Client Id for your Intercom application.
- `client_secret` string — Client Secret for your Intercom application.
- `activity_logs_time_step` integer — Set lower value in case of failing long running sync of Activity Logs stream.
- `sourceType` 'intercom', required
- SourceIp2whois
- `api_key` string — Your API Key. See here.
- `domain` string — Domain name. See here.
- `sourceType` 'ip2whois', required
- SourceIterable
- `api_key` string, required — Iterable API Key. See the docs for more information on how to obtain this key.
- `start_date` string, date-time, required — The date from which you'd like to replicate data for Iterable, in the format YYYY-MM-DDT00:00:00Z. All data generated after this date will be replicated.
- `sourceType` 'iterable', required
- SourceJira
- `api_token` string, required — Jira API Token. See the docs for more information on how to generate this key. API Token is used for Authorization to your account by BasicAuth.
- `domain` string, required — The Domain for your Jira account, e.g. airbyteio.atlassian.net, airbyteio.jira.com, jira.your-domain.com
- `email` string, required — The user email for your Jira account which you used to generate the API token. This field is used for Authorization to your account by BasicAuth.
- `projects` string[] — List of Jira project keys to replicate data for, or leave it empty if you want to replicate data for all projects.
- `start_date` string, date-time — The date from which you want to replicate data from Jira, use the format YYYY-MM-DDT00:00:00Z. Note that this field only applies to certain streams, and only data generated on or after the start date will be replicated. Or leave it empty if you want to replicate all data. For more information, refer to the documentation.
- `lookback_window_minutes` integer — When set to N, the connector will always refresh resources created within the past N minutes. By default, updated objects that are not newly created are not incrementally synced.
- `enable_experimental_streams` boolean — Allow the use of experimental streams which rely on undocumented Jira API endpoints. See https://docs.airbyte.com/integrations/sources/jira#experimental-tables for more info.
- `sourceType` 'jira', required
- SourceJotform
- `api_key` string, required
- `end_date` string, date-time, required
- `start_date` string, date-time, required
- `api_endpoint` union, required
- object
- `url_prefix` 'Standard' | 'EU' | 'HIPAA', required — You can access our API through the following URLs - Standard API Usage (Use the default API URL - https://api.jotform.com), For EU (Use the EU API URL - https://eu-api.jotform.com), For HIPAA (Use the HIPAA API URL - https://hipaa-api.jotform.com)
- `api_endpoint` 'basic'
- object
- `api_endpoint` 'enterprise'
- `enterprise_url` string, required — Upgrade to Enterprise to make your API url your-domain.com/API or subdomain.jotform.com/API instead of api.jotform.com
- `sourceType` 'jotform', required
- SourceK6Cloud
- `api_token` string, required — Your API Token. See here. The key is case sensitive.
- `sourceType` 'k6-cloud', required
- SourceKissmetrics
- `username` string, required
- `password` string
- `sourceType` 'kissmetrics', required
- SourceKlarna
- `region` 'eu' | 'na' | 'oc', required — Base url region (For playground eu https://docs.klarna.com/klarna-payments/api/payments-api/#tag/API-URLs). Supported 'eu', 'na', 'oc'
- `playground` boolean, required — Propertie defining if connector is used against playground or production environment
- `username` string, required — Consists of your Merchant ID (eid) - a unique number that identifies your e-store, combined with a random string (https://developers.klarna.com/api/#authentication)
- `password` string, required — A string which is associated with your Merchant ID and is used to authorize use of Klarna's APIs (https://developers.klarna.com/api/#authentication)
- `sourceType` 'klarna', required
- SourceKlausApi
- `api_key` string, required — API access key used to retrieve data from the KLAUS API.
- `account` integer, required — getting data by account
- `workspace` integer, required — getting data by workspace
- `start_date` string, date-time — Start getting data from that date.
- `sourceType` 'klaus-api', required
- SourceKlaviyo
- `api_key` string, required — Klaviyo API Key. See our docs if you need help finding this key.
- `start_date` string, date-time — UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated. This field is optional - if not provided, all data will be replicated.
- `disable_fetching_predictive_analytics` boolean — Certain streams like the profiles stream can retrieve predictive analytics data from Klaviyo's API. However, at high volume, this can lead to service availability issues on the API which can be improved by not fetching this field. WARNING: Enabling this setting will stop the "predictive_analytics" column from being populated in your downstream destination.
- `sourceType` 'klaviyo', required
- SourceKyve
- `pool_ids` string, required — The IDs of the KYVE storage pool you want to archive. (Comma separated)
- `start_ids` string, required — The start-id defines, from which bundle id the pipeline should start to extract the data. (Comma separated)
- `url_base` string, required — URL to the KYVE Chain API.
- `sourceType` 'kyve', required
- SourceLaunchdarkly
- `access_token` string, required — Your Access token. See here.
- `sourceType` 'launchdarkly', required
- SourceLeadfeeder
- `api_token` string, required
- `start_date` string, date-time, required
- `sourceType` 'leadfeeder', required
- SourceLemlist
- `api_key` string, required — Lemlist API key,
- `sourceType` 'lemlist', required
- SourceLeverHiring
- `credentials` union — Choose how to authenticate to Lever Hiring.
- object
- `auth_type` 'Client'
- `client_id` string — The Client ID of your Lever Hiring developer application.
- `client_secret` string — The Client Secret of your Lever Hiring developer application.
- `refresh_token` string, required — The token for obtaining new access token.
- object
- `auth_type` 'Api Key'
- `api_key` string, required — The Api Key of your Lever Hiring account.
- `start_date` string, required — UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated. Note that it will be used only in the following incremental streams: comments, commits, and issues.
- `environment` 'Production' | 'Sandbox' — The environment in which you'd like to replicate data for Lever. This is used to determine which Lever API endpoint to use.
- `sourceType` 'lever-hiring', required
- SourceLinkedinAds
- `credentials` union
- object
- `auth_method` 'oAuth2.0'
- `client_id` string, required — The client ID of your developer application. Refer to our documentation for more information.
- `client_secret` string, required — The client secret of your developer application. Refer to our documentation for more information.
- `refresh_token` string, required — The key to refresh the expired access token. Refer to our documentation for more information.
- object
- `auth_method` 'access_token'
- `access_token` string, required — The access token generated for your developer application. Refer to our documentation for more information.
- `start_date` string, date, required — UTC date in the format YYYY-MM-DD. Any data before this date will not be replicated.
- `lookback_window` integer — How far into the past to look for records. (in days)
- `account_ids` integer[] — Specify the account IDs to pull data from, separated by a space. Leave this field empty if you want to pull the data from all accounts accessible by the authenticated user. See the LinkedIn docs to locate these IDs.
- `ad_analytics_reports` object[]
- `name` string, required — The name for the custom report.
- `pivot_by` 'COMPANY' | 'ACCOUNT' | 'SHARE' | 'CAMPAIGN' | 'CREATIVE' | 'CAMPAIGN_GROUP' | 'CONVERSION' | 'CONVERSATION_NODE' | 'CONVERSATION_NODE_OPTION_INDEX' | 'SERVING_LOCATION' | 'CARD_INDEX' | 'MEMBER_COMPANY_SIZE' | 'MEMBER_INDUSTRY' | 'MEMBER_SENIORITY' | 'MEMBER_JOB_TITLE' | 'MEMBER_JOB_FUNCTION' | 'MEMBER_COUNTRY_V2' | 'MEMBER_REGION_V2' | 'MEMBER_COMPANY' | 'PLACEMENT_NAME' | 'IMPRESSION_DEVICE_TYPE', required — Choose a category to pivot your analytics report around. This selection will organize your data based on the chosen attribute, allowing you to analyze trends and performance from different perspectives.
- `time_granularity` 'ALL' | 'DAILY' | 'MONTHLY' | 'YEARLY', required — Choose how to group the data in your report by time. The options are:
- 'ALL': A single result summarizing the entire time range.
- 'DAILY': Group results by each day.
- 'MONTHLY': Group results by each month.
- 'YEARLY': Group results by each year.
Selecting a time grouping helps you analyze trends and patterns over different time periods.
- `sourceType` 'linkedin-ads', required
- SourceLinkedinPages
- `credentials` union
- object
- `client_id` string, required — The client ID of the LinkedIn developer application.
- `auth_method` 'oAuth2.0'
- `client_secret` string, required — The client secret of the LinkedIn developer application.
- `refresh_token` string, required — The token value generated using the LinkedIn Developers OAuth Token Tools. See the docs to obtain yours.
- object
- `auth_method` 'access_token'
- `access_token` string, required — The token value generated using the LinkedIn Developers OAuth Token Tools. See the docs to obtain yours.
- `org_id` string, required — Specify the Organization ID
- `start_date` string, date-time — Start date for getting metrics per time period. Must be atmost 12 months before the request date (UTC) and atleast 2 days prior to the request date (UTC). See https://bit.ly/linkedin-pages-date-rules {{ "\n" }} {{ response.errorDetails }}
- `time_granularity_type` 'DAY' | 'MONTH' — Granularity of the statistics for metrics per time period. Must be either "DAY" or "MONTH"
- `sourceType` 'linkedin-pages', required
- SourceLinnworks
- `application_id` string, required — Linnworks Application ID
- `application_secret` string, required — Linnworks Application Secret
- `token` string, required
- `start_date` string, date-time, required — UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.
- `sourceType` 'linnworks', required
- SourceLob
- `api_key` string, required — API key to use for authentication. You can find your account's API keys in your Dashboard Settings at https://dashboard.lob.com/settings/api-keys.
- `start_date` string, date-time, required
- `limit` string — Max records per page limit
- `sourceType` 'lob', required
- SourceLokalise
- `api_key` string, required — Lokalise API Key with read-access. Available at Profile settings > API tokens. See here.
- `project_id` string, required — Lokalise project ID. Available at Project Settings > General.
- `sourceType` 'lokalise', required
- SourceLooker
- `client_id` string, required — The Client ID is first part of an API3 key that is specific to each Looker user. See the docs for more information on how to generate this key.
- `client_secret` string, required — The Client Secret is second part of an API3 key.
- `domain` string, required — Domain for your Looker account, e.g. airbyte.cloud.looker.com,looker.[clientname].com,IP address
- `run_look_ids` string[] — The IDs of any Looks to run
- `sourceType` 'looker', required
- SourceLuma
- `api_key` string, required — Get your API key on lu.ma Calendars dashboard → Settings.
- `sourceType` 'luma', required
- SourceMailchimp
- `credentials` union
- object
- `auth_type` 'oauth2.0', required
- `client_id` string — The Client ID of your OAuth application.
- `client_secret` string — The Client Secret of your OAuth application.
- `access_token` string, required — An access token generated using the above client ID and secret.
- object
- `auth_type` 'apikey', required
- `apikey` string, required — Mailchimp API Key. See the docs for information on how to generate this key.
- `start_date` string, date-time — The date from which you want to start syncing data for Incremental streams. Only records that have been created or modified since this date will be synced. If left blank, all data will by synced.
- `sourceType` 'mailchimp', required
- SourceMailerlite
- `api_token` string, required — Your API Token. See here.
- `sourceType` 'mailerlite', required
- SourceMailgun
- `private_key` string, required — Primary account API key to access your Mailgun data.
- `domain_region` 'US' | 'EU' — Domain region code. 'EU' or 'US' are possible values. The default is 'US'.
- `start_date` string, date-time — UTC date and time in the format 2020-10-01 00:00:00. Any data before this date will not be replicated. If omitted, defaults to 3 days ago.
- `sourceType` 'mailgun', required
- SourceMailjetMail
- `api_key` string, required — Your API Key. See here.
- `api_key_secret` string, required — Your API Secret Key. See here.
- `sourceType` 'mailjet-mail', required
- SourceMailjetSms
- `end_date` integer — Retrieve SMS messages created before the specified timestamp. Required format - Unix timestamp.
- `start_date` integer — Retrieve SMS messages created after the specified timestamp. Required format - Unix timestamp.
- `token` string, required — Your access token. See here.
- `sourceType` 'mailjet-sms', required
- SourceMarketo
- `domain_url` string, required — Your Marketo Base URL. See the docs for info on how to obtain this.
- `client_id` string, required — The Client ID of your Marketo developer application. See the docs for info on how to obtain this.
- `client_secret` string, required — The Client Secret of your Marketo developer application. See the docs for info on how to obtain this.
- `start_date` string, date-time, required — UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.
- `sourceType` 'marketo', required
- SourceMetabase
- `instance_api_url` string, required — URL to your metabase instance API
- `username` string, required
- `password` string
- `session_token` string — To generate your session token, you need to run the following command: ``` curl -X POST \ -H "Content-Type: application/json" \ -d '{"username": "person@metabase.com", "password": "fakepassword"}' \ http://localhost:3000/api/session ``` Then copy the value of the `id` field returned by a successful call to that API. Note that by default, sessions are good for 14 days and needs to be regenerated.
- `sourceType` 'metabase', required
- SourceMicrosoftDataverse
- `url` string, required — URL to Microsoft Dataverse API
- `tenant_id` string, required — Tenant Id of your Microsoft Dataverse Instance
- `client_id` string, required — App Registration Client Id
- `client_secret_value` string, required — App Registration Client Secret
- `odata_maxpagesize` integer — Max number of results per page. Default=5000
- `sourceType` 'microsoft-dataverse', required
- SourceMicrosoftOnedrive — SourceMicrosoftOneDriveSpec class for Microsoft OneDrive Source Specification. This class combines the authentication details with additional configuration for the OneDrive API.
- `start_date` string, date-time — UTC date and time in the format 2017-01-25T00:00:00.000000Z. Any file modified before this date will not be replicated.
- `streams` object[], required — Each instance of this configuration defines a stream. Use this to define which files belong in the stream, their format, and how they should be parsed and validated. When sending data to warehouse destination such as Snowflake or BigQuery, each stream is a separate table.
- `name` string, required — The name of the stream.
- `globs` string[] — The pattern used to specify which files should be selected from the file system. For more information on glob pattern matching look here.
- `validation_policy` 'Emit Record' | 'Skip Record' | 'Wait for Discover' — The name of the validation policy that dictates sync behavior when a record does not adhere to the stream schema.
- `input_schema` string — The schema that will be used to validate records extracted from the file. This will override the stream schema that is auto-detected from incoming files.
- `days_to_sync_if_history_is_full` integer — When the state history of the file store is full, syncs will only read files that were last modified in the provided day range.
- `format` union, required — The configuration options that are used to alter how to read incoming files that deviate from the standard formatting.
- object
- `filetype` 'avro', required
- `double_as_string` boolean — Whether to convert double fields to strings. This is recommended if you have decimal numbers with a high degree of precision because there can be a loss precision when handling floating point numbers.
- object
- `filetype` 'csv', required
- `delimiter` string — The character delimiting individual cells in the CSV data. This may only be a 1-character string. For tab-delimited data enter '\t'.
- `quote_char` string — The character used for quoting CSV values. To disallow quoting, make this field blank.
- `escape_char` string — The character used for escaping special characters. To disallow escaping, leave this field blank.
- `encoding` string — The character encoding of the CSV data. Leave blank to default to UTF8. See list of python encodings for allowable options.
- `double_quote` boolean — Whether two quotes in a quoted CSV value denote a single quote in the data.
- `null_values` string[] — A set of case-sensitive strings that should be interpreted as null values. For example, if the value 'NA' should be interpreted as null, enter 'NA' in this field.
- `strings_can_be_null` boolean — Whether strings can be interpreted as null values. If true, strings that match the null_values set will be interpreted as null. If false, strings that match the null_values set will be interpreted as the string itself.
- `skip_rows_before_header` integer — The number of rows to skip before the header row. For example, if the header row is on the 3rd row, enter 2 in this field.
- `skip_rows_after_header` integer — The number of rows to skip after the header row.
- `header_definition` union — How headers will be defined. `User Provided` assumes the CSV does not have a header row and uses the headers provided and `Autogenerated` assumes the CSV does not have a header row and the CDK will generate headers using for `f{i}` where `i` is the index starting from 0. Else, the default behavior is to use the header from the CSV file. If a user wants to autogenerate or provide column names for a CSV having headers, they can skip rows.
- …
- `true_values` string[] — A set of case-sensitive strings that should be interpreted as true values.
- `false_values` string[] — A set of case-sensitive strings that should be interpreted as false values.
- `ignore_errors_on_fields_mismatch` boolean — Whether to ignore errors that occur when the number of fields in the CSV does not match the number of columns in the schema.
- object
- `filetype` 'jsonl', required
- object
- `filetype` 'parquet', required
- `decimal_as_float` boolean — Whether to convert decimal fields to floats. There is a loss of precision when converting decimals to floats, so this is not recommended.
- object — Extract text from document formats (.pdf, .docx, .md, .pptx) and emit as one record per file.
- `filetype` 'unstructured', required
- `skip_unprocessable_files` boolean — If true, skip files that cannot be parsed and pass the error message along as the _ab_source_file_parse_error field. If false, fail the sync.
- `strategy` 'auto' | 'fast' | 'ocr_only' | 'hi_res' — The strategy used to parse documents. `fast` extracts text directly from the document which doesn't work for all files. `ocr_only` is more reliable, but slower. `hi_res` is the most reliable, but requires an API key and a hosted instance of unstructured and can't be used with local mode. See the unstructured.io documentation for more details: https://unstructured-io.github.io/unstructured/core/partition.html#partition-pdf
- `processing` object — Process files locally, supporting `fast` and `ocr` modes. This is the default option.
- …
- `schemaless` boolean — When enabled, syncs will not validate or structure records against the stream's schema.
- `credentials` union, required — Credentials for connecting to the One Drive API
- object — OAuthCredentials class to hold authentication details for Microsoft OAuth authentication. This class uses pydantic for data validation and settings management.
- `auth_type` 'Client'
- `tenant_id` string, required — Tenant ID of the Microsoft OneDrive user
- `client_id` string, required — Client ID of your Microsoft developer application
- `client_secret` string, required — Client Secret of your Microsoft developer application
- `refresh_token` string, required — Refresh Token of your Microsoft developer application
- object — ServiceCredentials class for service key authentication. This class is structured similarly to OAuthCredentials but for a different authentication method.
- `auth_type` 'Service'
- `tenant_id` string, required — Tenant ID of the Microsoft OneDrive user
- `user_principal_name` string, required — Special characters such as a period, comma, space, and the at sign (@) are converted to underscores (_). More details: https://learn.microsoft.com/en-us/sharepoint/list-onedrive-urls
- `client_id` string, required — Client ID of your Microsoft developer application
- `client_secret` string, required — Client Secret of your Microsoft developer application
- `drive_name` string — Name of the Microsoft OneDrive drive where the file(s) exist.
- `search_scope` 'ACCESSIBLE_DRIVES' | 'SHARED_ITEMS' | 'ALL' — Specifies the location(s) to search for files. Valid options are 'ACCESSIBLE_DRIVES' to search in the selected OneDrive drive, 'SHARED_ITEMS' for shared items the user has access to, and 'ALL' to search both.
- `folder_path` string — Path to a specific folder within the drives to search for files. Leave empty to search all folders of the drives. This does not apply to shared items.
- `sourceType` 'microsoft-onedrive', required
- SourceMicrosoftSharepoint — SourceMicrosoftSharePointSpec class for Microsoft SharePoint Source Specification. This class combines the authentication details with additional configuration for the SharePoint API.
- `start_date` string, date-time — UTC date and time in the format 2017-01-25T00:00:00.000000Z. Any file modified before this date will not be replicated.
- `streams` object[], required — Each instance of this configuration defines a stream. Use this to define which files belong in the stream, their format, and how they should be parsed and validated. When sending data to warehouse destination such as Snowflake or BigQuery, each stream is a separate table.
- `name` string, required — The name of the stream.
- `globs` string[] — The pattern used to specify which files should be selected from the file system. For more information on glob pattern matching look here.
- `validation_policy` 'Emit Record' | 'Skip Record' | 'Wait for Discover' — The name of the validation policy that dictates sync behavior when a record does not adhere to the stream schema.
- `input_schema` string — The schema that will be used to validate records extracted from the file. This will override the stream schema that is auto-detected from incoming files.
- `days_to_sync_if_history_is_full` integer — When the state history of the file store is full, syncs will only read files that were last modified in the provided day range.
- `format` union, required — The configuration options that are used to alter how to read incoming files that deviate from the standard formatting.
- object
- `filetype` 'avro', required
- `double_as_string` boolean — Whether to convert double fields to strings. This is recommended if you have decimal numbers with a high degree of precision because there can be a loss precision when handling floating point numbers.
- object
- `filetype` 'csv', required
- `delimiter` string — The character delimiting individual cells in the CSV data. This may only be a 1-character string. For tab-delimited data enter '\t'.
- `quote_char` string — The character used for quoting CSV values. To disallow quoting, make this field blank.
- `escape_char` string — The character used for escaping special characters. To disallow escaping, leave this field blank.
- `encoding` string — The character encoding of the CSV data. Leave blank to default to UTF8. See list of python encodings for allowable options.
- `double_quote` boolean — Whether two quotes in a quoted CSV value denote a single quote in the data.
- `null_values` string[] — A set of case-sensitive strings that should be interpreted as null values. For example, if the value 'NA' should be interpreted as null, enter 'NA' in this field.
- `strings_can_be_null` boolean — Whether strings can be interpreted as null values. If true, strings that match the null_values set will be interpreted as null. If false, strings that match the null_values set will be interpreted as the string itself.
- `skip_rows_before_header` integer — The number of rows to skip before the header row. For example, if the header row is on the 3rd row, enter 2 in this field.
- `skip_rows_after_header` integer — The number of rows to skip after the header row.
- `header_definition` union — How headers will be defined. `User Provided` assumes the CSV does not have a header row and uses the headers provided and `Autogenerated` assumes the CSV does not have a header row and the CDK will generate headers using for `f{i}` where `i` is the index starting from 0. Else, the default behavior is to use the header from the CSV file. If a user wants to autogenerate or provide column names for a CSV having headers, they can skip rows.
- …
- `true_values` string[] — A set of case-sensitive strings that should be interpreted as true values.
- `false_values` string[] — A set of case-sensitive strings that should be interpreted as false values.
- `ignore_errors_on_fields_mismatch` boolean — Whether to ignore errors that occur when the number of fields in the CSV does not match the number of columns in the schema.
- object
- `filetype` 'jsonl', required
- object
- `filetype` 'parquet', required
- `decimal_as_float` boolean — Whether to convert decimal fields to floats. There is a loss of precision when converting decimals to floats, so this is not recommended.
- object — Extract text from document formats (.pdf, .docx, .md, .pptx) and emit as one record per file.
- `filetype` 'unstructured', required
- `skip_unprocessable_files` boolean — If true, skip files that cannot be parsed and pass the error message along as the _ab_source_file_parse_error field. If false, fail the sync.
- `strategy` 'auto' | 'fast' | 'ocr_only' | 'hi_res' — The strategy used to parse documents. `fast` extracts text directly from the document which doesn't work for all files. `ocr_only` is more reliable, but slower. `hi_res` is the most reliable, but requires an API key and a hosted instance of unstructured and can't be used with local mode. See the unstructured.io documentation for more details: https://unstructured-io.github.io/unstructured/core/partition.html#partition-pdf
- `processing` object — Process files locally, supporting `fast` and `ocr` modes. This is the default option.
- …
- object
- `filetype` 'excel', required
- `schemaless` boolean — When enabled, syncs will not validate or structure records against the stream's schema.
- `recent_n_files_to_read_for_schema_discovery` integer — The number of resent files which will be used to discover the schema for this stream.
- `credentials` union, required — Credentials for connecting to the One Drive API
- object — OAuthCredentials class to hold authentication details for Microsoft OAuth authentication. This class uses pydantic for data validation and settings management.
- `auth_type` 'Client'
- `tenant_id` string, required — Tenant ID of the Microsoft SharePoint user
- `client_id` string, required — Client ID of your Microsoft developer application
- `client_secret` string, required — Client Secret of your Microsoft developer application
- `refresh_token` string — Refresh Token of your Microsoft developer application
- object — ServiceCredentials class for service key authentication. This class is structured similarly to OAuthCredentials but for a different authentication method.
- `auth_type` 'Service'
- `tenant_id` string, required — Tenant ID of the Microsoft SharePoint user
- `user_principal_name` string, required — Special characters such as a period, comma, space, and the at sign (@) are converted to underscores (_). More details: https://learn.microsoft.com/en-us/sharepoint/list-onedrive-urls
- `client_id` string, required — Client ID of your Microsoft developer application
- `client_secret` string, required — Client Secret of your Microsoft developer application
- `search_scope` 'ACCESSIBLE_DRIVES' | 'SHARED_ITEMS' | 'ALL' — Specifies the location(s) to search for files. Valid options are 'ACCESSIBLE_DRIVES' for all SharePoint drives the user can access, 'SHARED_ITEMS' for shared items the user has access to, and 'ALL' to search both.
- `folder_path` string — Path to a specific folder within the drives to search for files. Leave empty to search all folders of the drives. This does not apply to shared items.
- `sourceType` 'microsoft-sharepoint', required
- SourceMicrosoftTeams
- `period` string, required — Specifies the length of time over which the Team Device Report stream is aggregated. The supported values are: D7, D30, D90, and D180.
- `credentials` union — Choose how to authenticate to Microsoft
- object
- `auth_type` 'Client'
- `tenant_id` string, required — A globally unique identifier (GUID) that is different than your organization name or domain. Follow these steps to obtain: open one of the Teams where you belong inside the Teams Application -> Click on the … next to the Team title -> Click on Get link to team -> Copy the link to the team and grab the tenant ID form the URL
- `client_id` string, required — The Client ID of your Microsoft Teams developer application.
- `client_secret` string, required — The Client Secret of your Microsoft Teams developer application.
- `refresh_token` string, required — A Refresh Token to renew the expired Access Token.
- object
- `auth_type` 'Token'
- `tenant_id` string, required — A globally unique identifier (GUID) that is different than your organization name or domain. Follow these steps to obtain: open one of the Teams where you belong inside the Teams Application -> Click on the … next to the Team title -> Click on Get link to team -> Copy the link to the team and grab the tenant ID form the URL
- `client_id` string, required — The Client ID of your Microsoft Teams developer application.
- `client_secret` string, required — The Client Secret of your Microsoft Teams developer application.
- `sourceType` 'microsoft-teams', required
- SourceMissive
- `api_key` string, required
- `limit` string — Max records per page limit
- `start_date` string, date-time, required
- `kind` 'group' | 'organization' — Kind parameter for `contact_groups` stream
- `sourceType` 'missive', required
- SourceMixmax
- `api_key` string, required
- `start_date` string, date-time, required
- `sourceType` 'mixmax', required
- SourceMixpanel
- `credentials` union, required — Choose how to authenticate to Mixpanel
- object
- `option_title` 'Service Account'
- `username` string, required — Mixpanel Service Account Username. See the docs for more information on how to obtain this.
- `secret` string, required — Mixpanel Service Account Secret. See the docs for more information on how to obtain this.
- `project_id` integer, required — Your project ID number. See the docs for more information on how to obtain this.
- object
- `option_title` 'Project Secret'
- `api_secret` string, required — Mixpanel project secret. See the docs for more information on how to obtain this.
- `attribution_window` integer — A period of time for attributing results to ads and the lookback period after those actions occur during which ad results are counted. Default attribution window is 5 days. (This value should be non-negative integer)
- `project_timezone` string — Time zone in which integer date times are stored. The project timezone may be found in the project settings in the Mixpanel console.
- `select_properties_by_default` boolean — Setting this config parameter to TRUE ensures that new properties on events and engage records are captured. Otherwise new properties will be ignored.
- `start_date` string, date-time — The date in the format YYYY-MM-DD. Any data before this date will not be replicated. If this option is not set, the connector will replicate data from up to one year ago by default.
- `end_date` string, date-time — The date in the format YYYY-MM-DD. Any data after this date will not be replicated. Left empty to always sync to most recent date
- `region` 'US' | 'EU' — The region of mixpanel domain instance either US or EU.
- `date_window_size` integer — Defines window size in days, that used to slice through data. You can reduce it, if amount of data in each window is too big for your environment. (This value should be positive integer)
- `page_size` integer — The number of records to fetch per request for the engage stream. Default is 1000. If you are experiencing long sync times with this stream, try increasing this value.
- `sourceType` 'mixpanel', required
- SourceMonday
- `credentials` union
- object
- `subdomain` string — Slug/subdomain of the account, or the first part of the URL that comes before .monday.com
- `auth_type` 'oauth2.0', required
- `client_id` string, required — The Client ID of your OAuth application.
- `client_secret` string, required — The Client Secret of your OAuth application.
- `access_token` string, required — Access Token for making authenticated requests.
- object
- `auth_type` 'api_token', required
- `api_token` string, required — API Token for making authenticated requests.
- `sourceType` 'monday', required
- SourceMongodbV2
- `database_config` union, required — Configures the MongoDB cluster type.
- object — MongoDB Atlas-hosted cluster configured as a replica set
- `cluster_type` 'ATLAS_REPLICA_SET', required
- `connection_string` string, required — The connection string of the cluster that you want to replicate.
- `database` string, required — The name of the MongoDB database that contains the collection(s) to replicate.
- `username` string, required — The username which is used to access the database.
- `password` string, required — The password associated with this username.
- `auth_source` string, required — The authentication source where the user information is stored. See https://www.mongodb.com/docs/manual/reference/connection-string/#mongodb-urioption-urioption.authSource for more details.
- `schema_enforced` boolean — When enabled, syncs will validate and structure records against the stream's schema.
- object — MongoDB self-hosted cluster configured as a replica set
- `cluster_type` 'SELF_MANAGED_REPLICA_SET', required
- `connection_string` string, required — The connection string of the cluster that you want to replicate. https://www.mongodb.com/docs/manual/reference/connection-string/#find-your-self-hosted-deployment-s-connection-string for more information.
- `database` string, required — The name of the MongoDB database that contains the collection(s) to replicate.
- `username` string — The username which is used to access the database.
- `password` string — The password associated with this username.
- `auth_source` string — The authentication source where the user information is stored.
- `schema_enforced` boolean — When enabled, syncs will validate and structure records against the stream's schema.
- `initial_waiting_seconds` integer — The amount of time the connector will wait when it launches to determine if there is new data to sync or not. Defaults to 300 seconds. Valid range: 120 seconds to 1200 seconds.
- `queue_size` integer — The size of the internal queue. This may interfere with memory consumption and efficiency of the connector, please be careful.
- `discover_sample_size` integer — The maximum number of documents to sample when attempting to discover the unique fields for a collection.
- `invalid_cdc_cursor_position_behavior` 'Fail sync' | 'Re-sync data' — Determines whether Airbyte should fail or re-sync data in case of an stale/invalid cursor value into the WAL. If 'Fail sync' is chosen, a user will have to manually reset the connection before being able to continue syncing data. If 'Re-sync data' is chosen, Airbyte will automatically trigger a refresh but could lead to higher cloud costs and data loss.
- `update_capture_mode` 'Lookup' | 'Post Image' — Determines how Airbyte looks up the value of an updated document. If 'Lookup' is chosen, the current value of the document will be read. If 'Post Image' is chosen, then the version of the document immediately after an update will be read. WARNING : Severe data loss will occur if this option is chosen and the appropriate settings are not set on your Mongo instance : https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre-and-post-images.
- `initial_load_timeout_hours` integer — The amount of time an initial load is allowed to continue for before catching up on CDC logs.
- `sourceType` 'mongodb-v2', required
- SourceMssql
- `host` string, required — The hostname of the database.
- `port` integer, required — The port of the database.
- `database` string, required — The name of the database.
- `schemas` string[] — The list of schemas to sync from. Defaults to user. Case sensitive.
- `username` string, required — The username which is used to access the database.
- `password` string, required — The password associated with the username.
- `jdbc_url_params` string — Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (example: key1=value1&key2=value2&key3=value3).
- `ssl_method` union — The encryption method which is used when communicating with the database.
- object — Data transfer will not be encrypted.
- `ssl_method` 'unencrypted', required
- object — Use the certificate provided by the server without verification. (For testing purposes only!)
- `ssl_method` 'encrypted_trust_server_certificate', required
- object — Verify and use the certificate provided by the server.
- `ssl_method` 'encrypted_verify_certificate', required
- `hostNameInCertificate` string — Specifies the host name of the server. The value of this property must match the subject property of the certificate.
- `certificate` string — certificate of the server, or of the CA that signed the server certificate
- `replication_method` union — Configures how data is extracted from the database.
- object — Recommended - Incrementally reads new inserts, updates, and deletes using the SQL Server's change data capture feature. This must be enabled on your database.
- `method` 'CDC', required
- `initial_waiting_seconds` integer — The amount of time the connector will wait when it launches to determine if there is new data to sync or not. Defaults to 300 seconds. Valid range: 120 seconds to 3600 seconds. Read about initial waiting time.
- `invalid_cdc_cursor_position_behavior` 'Fail sync' | 'Re-sync data' — Determines whether Airbyte should fail or re-sync data in case of an stale/invalid cursor value into the WAL. If 'Fail sync' is chosen, a user will have to manually reset the connection before being able to continue syncing data. If 'Re-sync data' is chosen, Airbyte will automatically trigger a refresh but could lead to higher cloud costs and data loss.
- `queue_size` integer — The size of the internal queue. This may interfere with memory consumption and efficiency of the connector, please be careful.
- `initial_load_timeout_hours` integer — The amount of time an initial load is allowed to continue for before catching up on CDC logs.
- object — Incrementally detects new inserts and updates using the cursor column chosen when configuring a connection (e.g. created_at, updated_at).
- `method` 'STANDARD', required
- `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use.
- object
- `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database
- object
- `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key
- `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel.
- `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections.
- `tunnel_user` string, required — OS-level username for logging into the jump server host.
- `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa )
- object
- `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication
- `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel.
- `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections.
- `tunnel_user` string, required — OS-level username for logging into the jump server host
- `tunnel_user_password` string, required — OS-level password for logging into the jump server host
- `sourceType` 'mssql', required
- SourceMux
- `username` string, required
- `password` string
- `start_date` string, date-time, required
- `playback_id` string — The playback id for your video asset shown in website details
- `sourceType` 'mux', required
- SourceMyHours
- `email` string, required — Your My Hours username
- `password` string, required — The password associated to the username
- `start_date` string, required — Start date for collecting time logs
- `logs_batch_size` integer — Pagination size used for retrieving logs in days
- `sourceType` 'my-hours', required
- SourceMysql
- `host` string, required — The host name of the database.
- `port` integer, required — The port to connect to.
- `database` string, required — The database name.
- `username` string, required — The username which is used to access the database.
- `password` string — The password associated with the username.
- `jdbc_url_params` string — Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (example: key1=value1&key2=value2&key3=value3). For more information read about JDBC URL parameters.
- `ssl` boolean — Encrypt data using SSL.
- `ssl_mode` union — SSL connection modes. Read more in the docs.
- object — Automatically attempt SSL connection. If the MySQL server does not support SSL, continue with a regular connection.
- `mode` 'preferred', required
- object — Always connect with SSL. If the MySQL server doesn’t support SSL, the connection will not be established. Certificate Authority (CA) and Hostname are not verified.
- `mode` 'required', required
- object — Always connect with SSL. Verifies CA, but allows connection even if Hostname does not match.
- `mode` 'verify_ca', required
- `ca_certificate` string, required — CA certificate
- `client_certificate` string — Client certificate (this is not a required field, but if you want to use it, you will need to add the Client key as well)
- `client_key` string — Client key (this is not a required field, but if you want to use it, you will need to add the Client certificate as well)
- `client_key_password` string — Password for keystorage. This field is optional. If you do not add it - the password will be generated automatically.
- object — Always connect with SSL. Verify both CA and Hostname.
- `mode` 'verify_identity', required
- `ca_certificate` string, required — CA certificate
- `client_certificate` string — Client certificate (this is not a required field, but if you want to use it, you will need to add the Client key as well)
- `client_key` string — Client key (this is not a required field, but if you want to use it, you will need to add the Client certificate as well)
- `client_key_password` string — Password for keystorage. This field is optional. If you do not add it - the password will be generated automatically.
- `replication_method` union, required — Configures how data is extracted from the database.
- object — Recommended - Incrementally reads new inserts, updates, and deletes using the MySQL binary log. This must be enabled on your database.
- `method` 'CDC', required
- `initial_waiting_seconds` integer — The amount of time the connector will wait when it launches to determine if there is new data to sync or not. Defaults to 300 seconds. Valid range: 120 seconds to 1200 seconds. Read about initial waiting time.
- `server_time_zone` string — Enter the configured MySQL server timezone. This should only be done if the configured timezone in your MySQL instance does not conform to IANNA standard.
- `invalid_cdc_cursor_position_behavior` 'Fail sync' | 'Re-sync data' — Determines whether Airbyte should fail or re-sync data in case of an stale/invalid cursor value into the WAL. If 'Fail sync' is chosen, a user will have to manually reset the connection before being able to continue syncing data. If 'Re-sync data' is chosen, Airbyte will automatically trigger a refresh but could lead to higher cloud costs and data loss.
- `initial_load_timeout_hours` integer — The amount of time an initial load is allowed to continue for before catching up on CDC logs.
- object — Incrementally detects new inserts and updates using the cursor column chosen when configuring a connection (e.g. created_at, updated_at).
- `method` 'STANDARD', required
- `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use.
- object
- `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database
- object
- `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key
- `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel.
- `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections.
- `tunnel_user` string, required — OS-level username for logging into the jump server host.
- `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa )
- object
- `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication
- `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel.
- `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections.
- `tunnel_user` string, required — OS-level username for logging into the jump server host
- `tunnel_user_password` string, required — OS-level password for logging into the jump server host
- `sourceType` 'mysql', required
- SourceN8n
- `host` string, required — Hostname of the n8n instance
- `api_key` string, required — Your API KEY. See here
- `sourceType` 'n8n', required
- SourceNasa
- `api_key` string, required — API access key used to retrieve data from the NASA APOD API.
- `concept_tags` boolean — Indicates whether concept tags should be returned with the rest of the response. The concept tags are not necessarily included in the explanation, but rather derived from common search tags that are associated with the description text. (Better than just pure text search.) Defaults to False.
- `count` integer — A positive integer, no greater than 100. If this is specified then `count` randomly chosen images will be returned in a JSON array. Cannot be used in conjunction with `date` or `start_date` and `end_date`.
- `end_date` string, date — Indicates that end of a date range. If `start_date` is specified without an `end_date` then `end_date` defaults to the current date.
- `start_date` string, date — Indicates the start of a date range. All images in the range from `start_date` to `end_date` will be returned in a JSON array. Must be after 1995-06-16, the first day an APOD picture was posted. There are no images for tomorrow available through this API.
- `thumbs` boolean — Indicates whether the API should return a thumbnail image URL for video files. If set to True, the API returns URL of video thumbnail. If an APOD is not a video, this parameter is ignored.
- `sourceType` 'nasa', required
- SourceNetsuite
- `realm` string, required — Netsuite realm e.g. 2344535, as for `production` or 2344535_SB1, as for the `sandbox`
- `consumer_key` string, required — Consumer key associated with your integration
- `consumer_secret` string, required — Consumer secret associated with your integration
- `token_key` string, required — Access token key
- `token_secret` string, required — Access token secret
- `object_types` string[] — The API names of the Netsuite objects you want to sync. Setting this speeds up the connection setup process by limiting the number of schemas that need to be retrieved from Netsuite.
- `start_datetime` string, required — Starting point for your data replication, in format of "YYYY-MM-DDTHH:mm:ssZ"
- `window_in_days` integer — The amount of days used to query the data with date chunks. Set smaller value, if you have lots of data.
- `sourceType` 'netsuite', required
- SourceNewsApi
- `api_key` string, required — API Key
- `search_query` string — Search query. See https://newsapi.org/docs/endpoints/everything for information.
- `search_in` string[] — Where to apply search query. Possible values are: title, description, content.
- `sources` string[] — Identifiers (maximum 20) for the news sources or blogs you want headlines from. Use the `/sources` endpoint to locate these programmatically or look at the sources index: https://newsapi.com/sources. Will override both country and category.
- `domains` string[] — A comma-seperated string of domains (eg bbc.co.uk, techcrunch.com, engadget.com) to restrict the search to.
- `exclude_domains` string[] — A comma-seperated string of domains (eg bbc.co.uk, techcrunch.com, engadget.com) to remove from the results.
- `start_date` string — A date and optional time for the oldest article allowed. This should be in ISO 8601 format.
- `end_date` string — A date and optional time for the newest article allowed. This should be in ISO 8601 format.
- `language` 'ar' | 'de' | 'en' | 'es' | 'fr' | 'he' | 'it' | 'nl' | 'no' | 'pt' | 'ru' | 'se' | 'ud' | 'zh' — The 2-letter ISO-639-1 code of the language you want to get headlines for. Possible options: ar de en es fr he it nl no pt ru se ud zh.
- `country` 'ae' | 'ar' | 'at' | 'au' | 'be' | 'bg' | 'br' | 'ca' | 'ch' | 'cn' | 'co' | 'cu' | 'cz' | 'de' | 'eg' | 'fr' | 'gb' | 'gr' | 'hk' | 'hu' | 'id' | 'ie' | 'il' | 'in' | 'it' | 'jp' | 'kr' | 'lt' | 'lv' | 'ma' | 'mx' | 'my' | 'ng' | 'nl' | 'no' | 'nz' | 'ph' | 'pl' | 'pt' | 'ro' | 'rs' | 'ru' | 'sa' | 'se' | 'sg' | 'si' | 'sk' | 'th' | 'tr' | 'tw' | 'ua' | 'us' | 've' | 'za', required — The 2-letter ISO 3166-1 code of the country you want to get headlines for. You can't mix this with the sources parameter.
- `category` 'business' | 'entertainment' | 'general' | 'health' | 'science' | 'sports' | 'technology', required — The category you want to get top headlines for.
- `sort_by` 'relevancy' | 'popularity' | 'publishedAt', required — The order to sort the articles in. Possible options: relevancy, popularity, publishedAt.
- `sourceType` 'news-api', required
- SourceNorthpassLms
- `api_key` string, required
- `sourceType` 'northpass-lms', required
- SourceNotion
- `start_date` string, date-time — UTC date and time in the format YYYY-MM-DDTHH:MM:SS.000Z. During incremental sync, any data generated before this date will not be replicated. If left blank, the start date will be set to 2 years before the present date.
- `credentials` union — Choose either OAuth (recommended for Airbyte Cloud) or Access Token. See our docs for more information.
- object
- `auth_type` 'OAuth2.0', required
- `client_id` string, required — The Client ID of your Notion integration. See our docs for more information.
- `client_secret` string, required — The Client Secret of your Notion integration. See our docs for more information.
- `access_token` string, required — The Access Token received by completing the OAuth flow for your Notion integration. See our docs for more information.
- object
- `auth_type` 'token', required
- `token` string, required — The Access Token for your private Notion integration. See the docs for more information on how to obtain this token.
- `sourceType` 'notion'
- SourceNylas
- `api_key` string, required
- `api_server` 'us' | 'eu', required
- `start_date` string, date-time, required
- `end_date` string, date-time, required
- `sourceType` 'nylas', required
- SourceNytimes
- `api_key` string, required — API Key
- `start_date` string, required — Start date to begin the article retrieval (format YYYY-MM)
- `end_date` string — End date to stop the article retrieval (format YYYY-MM)
- `period` 1 | 7 | 30, required — Period of time (in days)
- `share_type` 'facebook' — Share Type
- `sourceType` 'nytimes', required
- SourceOkta
- `credentials` union
- object
- `auth_type` 'oauth2.0', required
- `client_id` string, required — The Client ID of your OAuth application.
- `client_secret` string, required — The Client Secret of your OAuth application.
- `refresh_token` string, required — Refresh Token to obtain new Access Token, when it's expired.
- object
- `auth_type` 'oauth2.0_private_key', required
- `client_id` string, required — The Client ID of your OAuth application.
- `key_id` string, required — The key ID (kid).
- `private_key` string, required — The private key in PEM format
- `scope` string, required — The OAuth scope.
- object
- `auth_type` 'api_token', required
- `api_token` string, required — An Okta token. See the docs for instructions on how to generate it.
- `domain` string — The Okta domain. See the docs for instructions on how to find it.
- `start_date` string, date-time — UTC date and time in the format YYYY-MM-DDTHH:MM:SSZ. Any data before this date will not be replicated.
- `sourceType` 'okta', required
- SourceOmnisend
- `api_key` string, required — API Key
- `sourceType` 'omnisend', required
- SourceOnesignal
- `user_auth_key` string, required — OneSignal User Auth Key, see the docs for more information on how to obtain this key.
- `applications` object[], required — Applications keys, see the docs for more information on how to obtain this data
- `app_id` string, required
- `app_name` string
- `app_api_key` string, required
- `start_date` string, date-time, required — The date from which you'd like to replicate data for OneSignal API, in the format YYYY-MM-DDT00:00:00Z. All data generated after this date will be replicated.
- `outcome_names` string, required — Comma-separated list of names and the value (sum/count) for the returned outcome data. See the docs for more details
- `sourceType` 'onesignal', required
- SourceOpenDataDc
- `api_key` string, required
- `location` string — address or place or block
- `marid` string — A unique identifier (Master Address Repository).
- `sourceType` 'open-data-dc', required
- SourceOpenweather
- `appid` string, required — API KEY
- `lang` 'af' | 'al' | 'ar' | 'az' | 'bg' | 'ca' | 'cz' | 'da' | 'de' | 'el' | 'en' | 'eu' | 'fa' | 'fi' | 'fr' | 'gl' | 'he' | 'hi' | 'hr' | 'hu' | 'id' | 'it' | 'ja' | 'kr' | 'la' | 'lt' | 'mk' | 'no' | 'nl' | 'pl' | 'pt' | 'pt_br' | 'ro' | 'ru' | 'sv' | 'se' | 'sk' | 'sl' | 'sp' | 'es' | 'sr' | 'th' | 'tr' | 'ua' | 'uk' | 'vi' | 'zh_cn' | 'zh_tw' | 'zu' — You can use lang parameter to get the output in your language. The contents of the description field will be translated. See here for the list of supported languages.
- `lat` string, required — Latitude, decimal (-90; 90). If you need the geocoder to automatic convert city names and zip-codes to geo coordinates and the other way around, please use the OpenWeather Geocoding API
- `lon` string, required — Longitude, decimal (-180; 180). If you need the geocoder to automatic convert city names and zip-codes to geo coordinates and the other way around, please use the OpenWeather Geocoding API
- `only_current` boolean — True for particular day
- `units` 'standard' | 'metric' | 'imperial' — Units of measurement. standard, metric and imperial units are available. If you do not use the units parameter, standard units will be applied by default.
- `sourceType` 'openweather', required
- SourceOpsgenie
- `api_token` string, required — API token used to access the Opsgenie platform
- `endpoint` string, required — Service endpoint to use for API calls.
- `start_date` string — The date from which you'd like to replicate data from Opsgenie in the format of YYYY-MM-DDT00:00:00Z. All data generated after this date will be replicated. Note that it will be used only in the following incremental streams: issues.
- `sourceType` 'opsgenie', required
- SourceOracle
- `host` string, required — Hostname of the database.
- `port` integer, required — Port of the database. Oracle Corporations recommends the following port numbers: 1521 - Default listening port for client connections to the listener. 2484 - Recommended and officially registered listening port for client connections to the listener using TCP/IP with SSL
- `connection_data` union — Connect data that will be used for DB connection
- object — Use service name
- `connection_type` 'service_name'
- `service_name` string, required
- object — Use SID (Oracle System Identifier)
- `connection_type` 'sid'
- `sid` string, required
- `username` string, required — The username which is used to access the database.
- `password` string — The password associated with the username.
- `schemas` string[] — The list of schemas to sync from. Defaults to user. Case sensitive.
- `jdbc_url_params` string — Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (example: key1=value1&key2=value2&key3=value3).
- `encryption` union — The encryption method with is used when communicating with the database.
- object — Data transfer will not be encrypted.
- `encryption_method` 'unencrypted', required
- object — The native network encryption gives you the ability to encrypt database connections, without the configuration overhead of TCP/IP and SSL/TLS and without the need to open and listen on different ports.
- `encryption_method` 'client_nne', required
- `encryption_algorithm` 'AES256' | 'RC4_56' | '3DES168' — This parameter defines what encryption algorithm is used.
- object — Verify and use the certificate provided by the server.
- `encryption_method` 'encrypted_verify_certificate', required
- `ssl_certificate` string, required — Privacy Enhanced Mail (PEM) files are concatenated certificate containers frequently used in certificate installations.
- `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use.
- object
- `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database
- object
- `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key
- `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel.
- `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections.
- `tunnel_user` string, required — OS-level username for logging into the jump server host.
- `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa )
- object
- `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication
- `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel.
- `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections.
- `tunnel_user` string, required — OS-level username for logging into the jump server host
- `tunnel_user_password` string, required — OS-level password for logging into the jump server host
- `sourceType` 'oracle', required
- SourceOrb
- `api_key` string, required — Orb API Key, issued from the Orb admin console.
- `start_date` string, date-time, required — UTC date and time in the format 2022-03-01T00:00:00Z. Any data with created_at before this data will not be synced. For Subscription Usage, this becomes the `timeframe_start` API parameter.
- `end_date` string — UTC date and time in the format 2022-03-01T00:00:00Z. Any data with created_at after this data will not be synced. For Subscription Usage, this becomes the `timeframe_start` API parameter.
- `lookback_window_days` integer — When set to N, the connector will always refresh resources created within the past N days. By default, updated objects that are not newly created are not incrementally synced.
- `string_event_properties_keys` string[] — Property key names to extract from all events, in order to enrich ledger entries corresponding to an event deduction.
- `numeric_event_properties_keys` string[] — Property key names to extract from all events, in order to enrich ledger entries corresponding to an event deduction.
- `subscription_usage_grouping_key` string — Property key name to group subscription usage by.
- `plan_id` string — Orb Plan ID to filter subscriptions that should have usage fetched.
- `sourceType` 'orb', required
- SourceOrbit
- `api_token` string, required — Authorizes you to work with Orbit workspaces associated with the token.
- `workspace` string, required — The unique name of the workspace that your API token is associated with.
- `start_date` string — Date in the format 2022-06-26. Only load members whose last activities are after this date.
- `sourceType` 'orbit', required
- SourceOura
- `api_key` string, required — API Key
- `start_datetime` string, date-time — Start datetime to sync from. Default is current UTC datetime minus 1 day.
- `end_datetime` string, date-time — End datetime to sync until. Default is current UTC datetime.
- `sourceType` 'oura', required
- SourceOutbrainAmplify
- `credentials` union, required — Credentials for making authenticated requests requires either username/password or access_token.
- object
- `type` 'access_token', required
- `access_token` string, required — Access Token for making authenticated requests.
- object
- `type` 'username_password', required
- `username` string, required — Add Username for authentication.
- `password` string, required — Add Password for authentication.
- `report_granularity` 'daily' | 'weekly' | 'monthly' — The granularity used for periodic data in reports. See the docs.
- `geo_location_breakdown` 'country' | 'region' | 'subregion' — The granularity used for geo location data in reports.
- `start_date` string, required — Date in the format YYYY-MM-DD eg. 2017-01-25. Any data before this date will not be replicated.
- `end_date` string — Date in the format YYYY-MM-DD.
- `sourceType` 'outbrain-amplify', required
- SourceOutreach
- `client_id` string, required — The Client ID of your Outreach developer application.
- … truncated; see the full OpenAPI document linked below
## Other responses
- `403` — Not allowed
- `404` — Not found
## Changes
- **2024-10-23** `fae855ef1b76` — 2 breaking, 4 info
- the response property `configuration/oneOf[subschema #35: source-chameleon -> subschema #42: source-chameleon]/end_date` became optional for the status `200`
- added `#/components/schemas/source-activecampaign, #/components/schemas/source-apple-search-ads, #/components/schemas/source-ashby, #/components/schemas/source-bugsnag, #/components/schemas/source-callrail, #/components/schemas/source-campaign-monitor, #/components/schemas/source-care-quality-commission, #/components/schemas/source-chargify, #/components/schemas/source-circleci, #/components/schemas/source-coassemble, #/components/schemas/source-convertkit, #/components/schemas/source-copper, #/components/schemas/source-countercyclical, #/components/schemas/source-drift, #/components/schemas/source-drip, #/components/schemas/source-easypost, #/components/schemas/source-employment-hero, #/components/schemas/source-factorial, #/components/schemas/source-flexport, #/components/schemas/source-free-agent-connector, #/components/schemas/source-freshservice, #/components/schemas/source-gmail, #/components/schemas/source-gocardless, #/components/schemas/source-gong, #/components/schemas/source-google-calendar, #/components/schemas/source-gorgias, #/components/schemas/source-gutendex, #/components/schemas/source-illumina-basespace, #/components/schemas/source-incident-io, #/components/schemas/source-klaus-api, #/components/schemas/source-mailerlite, #/components/schemas/source-mailjet-mail, #/components/schemas/source-microsoft-dataverse, #/components/schemas/source-missive, #/components/schemas/source-mixmax, #/components/schemas/source-mux, #/components/schemas/source-n8n, #/components/schemas/source-nasa, #/components/schemas/source-news-api, #/components/schemas/source-open-data-dc, #/components/schemas/source-openweather, #/components/schemas/source-opsgenie, #/components/schemas/source-oura, #/components/schemas/source-pardot, #/components/schemas/source-persona, #/components/schemas/source-pivotal-tracker, #/components/schemas/source-plaid, #/components/schemas/source-plausible, #/components/schemas/source-primetric, #/components/schemas/source-quickbooks, #/components/schemas/source-rd-station-marketing, #/components/schemas/source-referralhero, #/components/schemas/source-reply-io, #/components/schemas/source-revenuecat, #/components/schemas/source-rootly, #/components/schemas/source-safetyculture, #/components/schemas/source-sage-hr, #/components/schemas/source-segment, #/components/schemas/source-sharetribe, #/components/schemas/source-sigma-computing, #/components/schemas/source-simplecast, #/components/schemas/source-simplesat, #/components/schemas/source-solarwinds-service-desk, #/components/schemas/source-statsig, #/components/schemas/source-statuspage, #/components/schemas/source-testrail, #/components/schemas/source-thinkific, #/components/schemas/source-timely, #/components/schemas/source-todoist, #/components/schemas/source-unleash, #/components/schemas/source-visma-economic, #/components/schemas/source-vitally, #/components/schemas/source-waiteraid, #/components/schemas/source-weatherstack, #/components/schemas/source-workable, #/components/schemas/source-workflowmax, #/components/schemas/source-workramp, #/components/schemas/source-wrike, #/components/schemas/source-xsolla, #/components/schemas/source-yahoo-finance-price, #/components/schemas/source-you-need-a-budget-ynab, #/components/schemas/source-zapier-supported-storage, #/components/schemas/source-zenefits` to the `configuration` response property `oneOf` list for the response status `200`
- added the new optional request property `configuration/oneOf[subschema #10: source-amplitude -> subschema #11: source-amplitude]/active_users_group_by_country`
- the request property `configuration/oneOf[subschema #35: source-chameleon -> subschema #42: source-chameleon]/end_date` became optional
- …2 more
[Change history](https://skmtc.dev/airbytehq/apis/airbyte-api/changes/sources/:sourceId/patch.md)
---
[API](https://skmtc.dev/airbytehq/apis/airbyte-api.md) · [All operations](https://skmtc.dev/airbytehq/apis/airbyte-api/llms.txt) · [OpenAPI document](https://skmtc-service-production.skmtc.workers.dev/v1/apis/airbytehq/airbyte-api/revisions/fae855ef1b76/schema)