--- title: "Create a destination" method: POST path: "/destinations" tags: ["Destinations"] --- # Create a destination `POST /destinations` Creates a destination given a name, workspace id, and a json blob containing the configuration for the source. ## Request body - DestinationCreateRequest - `name` string, required — Name of the destination e.g. dev-mysql-instance. - `definitionId` string, uuid — The UUID of the connector definition. One of configuration.destinationType or definitionId must be provided. - `workspaceId` string, uuid, required - `configuration` union, required — The values required to configure the destination. - DestinationGoogleSheets - `spreadsheet_id` string, required — The link to your spreadsheet. See this guide for more details. - `credentials` object, required — Google API Credentials for connecting to Google Sheets and Google Drive APIs - `client_id` string, required — The Client ID of your Google Sheets developer application. - `client_secret` string, required — The Client Secret of your Google Sheets developer application. - `refresh_token` string, required — The token for obtaining new access token. - `destinationType` 'google-sheets', required - DestinationAstra — The configuration model for the Vector DB based destinations. This model is used to generate the UI for the destination configuration, as well as to provide type safety for the configuration passed to the destination. The configuration model is composed of four parts: * Processing configuration * Embedding configuration * Indexing configuration * Advanced configuration Processing, embedding and advanced configuration are provided by this base class, while the indexing configuration is provided by the destination connector in the sub class. - `embedding` union, required — Embedding configuration - object — Use the OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `mode` 'openai', required - `openai_key` string, required - object — Use the Cohere API to embed text. - `mode` 'cohere', required - `cohere_key` string, required - object — Use a fake embedding made out of random vectors with 1536 embedding dimensions. This is useful for testing the data pipeline without incurring any costs. - `mode` 'fake', required - object — Use the Azure-hosted OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `mode` 'azure_openai', required - `openai_key` string, required — The API key for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `api_base` string, required — The base URL for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `deployment` string, required — The deployment for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - object — Use a service that's compatible with the OpenAI API to embed text. - `mode` 'openai_compatible', required - `api_key` string - `base_url` string, required — The base URL for your OpenAI-compatible service - `model_name` string — The name of the model to use for embedding - `dimensions` integer, required — The number of dimensions the embedding model is generating - `processing` object, required - `chunk_size` integer, required — Size of chunks in tokens to store in vector store (make sure it is not too big for the context if your LLM) - `chunk_overlap` integer — Size of overlap between chunks in tokens to store in vector store to better capture relevant context - `text_fields` string[] — List of fields in the record that should be used to calculate the embedding. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered text fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. - `metadata_fields` string[] — List of fields in the record that should be stored as metadata. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered metadata fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. When specifying nested paths, all matching values are flattened into an array set to a field named by the path. - `text_splitter` union — Split text fields into chunks based on the specified method. - object — Split the text by the list of separators until the chunk size is reached, using the earlier mentioned separators where possible. This is useful for splitting text fields by paragraphs, sentences, words, etc. - `mode` 'separator', required - `separators` string[] — List of separator strings to split text fields by. The separator itself needs to be wrapped in double quotes, e.g. to split by the dot character, use ".". To split by a newline, use "\n". - `keep_separator` boolean — Whether to keep the separator in the resulting chunks - object — Split the text by Markdown headers down to the specified header level. If the chunk size fits multiple sections, they will be combined into a single chunk. - `mode` 'markdown', required - `split_level` integer — Level of markdown headers to split text fields by. Headings down to the specified level will be used as split points - object — Split the text by suitable delimiters based on the programming language. This is useful for splitting code into chunks. - `mode` 'code', required - `language` 'cpp' | 'go' | 'java' | 'js' | 'php' | 'proto' | 'python' | 'rst' | 'ruby' | 'rust' | 'scala' | 'swift' | 'markdown' | 'latex' | 'html' | 'sol', required — Split code in suitable places based on the programming language - `field_name_mappings` object[] — List of fields to rename. Not applicable for nested fields, but can be used to rename fields already flattened via dot notation. - `from_field` string, required — The field name in the source - `to_field` string, required — The field name to use in the destination - `omit_raw_text` boolean — Do not store the text that gets embedded along with the vector and the metadata in the destination. If set to true, only the vector and the metadata will be stored - in this case raw text for LLM use cases needs to be retrieved from another source. - `indexing` object, required — Astra DB gives developers the APIs, real-time data and ecosystem integrations to put accurate RAG and Gen AI apps with fewer hallucinations in production. - `astra_db_app_token` string, required — The application token authorizes a user to connect to a specific Astra DB database. It is created when the user clicks the Generate Token button on the Overview tab of the Database page in the Astra UI. - `astra_db_endpoint` string, required — The endpoint specifies which Astra DB database queries are sent to. It can be copied from the Database Details section of the Overview tab of the Database page in the Astra UI. - `astra_db_keyspace` string, required — Keyspaces (or Namespaces) serve as containers for organizing data within a database. You can create a new keyspace uisng the Data Explorer tab in the Astra UI. The keyspace default_keyspace is created for you when you create a Vector Database in Astra DB. - `collection` string, required — Collections hold data. They are analagous to tables in traditional Cassandra terminology. This tool will create the collection with the provided name automatically if it does not already exist. Alternatively, you can create one thorugh the Data Explorer tab in the Astra UI. - `destinationType` 'astra', required - DestinationAwsDatalake - `aws_account_id` string — target aws account id - `credentials` union, required — Choose How to Authenticate to AWS. - object - `credentials_title` 'IAM Role', required — Name of the credentials - `role_arn` string, required — Will assume this role to write data to s3 - object - `credentials_title` 'IAM User', required — Name of the credentials - `aws_access_key_id` string, required — AWS User Access Key Id - `aws_secret_access_key` string, required — Secret Access Key - `region` '' | 'af-south-1' | 'ap-east-1' | 'ap-northeast-1' | 'ap-northeast-2' | 'ap-northeast-3' | 'ap-south-1' | 'ap-south-2' | 'ap-southeast-1' | 'ap-southeast-2' | 'ap-southeast-3' | 'ap-southeast-4' | 'ca-central-1' | 'ca-west-1' | 'cn-north-1' | 'cn-northwest-1' | 'eu-central-1' | 'eu-central-2' | 'eu-north-1' | 'eu-south-1' | 'eu-south-2' | 'eu-west-1' | 'eu-west-2' | 'eu-west-3' | 'il-central-1' | 'me-central-1' | 'me-south-1' | 'sa-east-1' | 'us-east-1' | 'us-east-2' | 'us-gov-east-1' | 'us-gov-west-1' | 'us-west-1' | 'us-west-2', required — The region of the S3 bucket. See here for all region codes. - `bucket_name` string, required — The name of the S3 bucket. Read more here. - `bucket_prefix` string — S3 prefix - `lakeformation_database_name` string, required — The default database this destination will use to create tables in per stream. Can be changed per connection by customizing the namespace. - `lakeformation_database_default_tag_key` string — Add a default tag key to databases created by this destination - `lakeformation_database_default_tag_values` string — Add default values for the `Tag Key` to databases created by this destination. Comma separate for multiple values. - `lakeformation_governed_tables` boolean — Whether to create tables as LF governed tables. - `format` union — Format of the data output. - object - `format_type` 'JSONL', required - `compression_codec` 'UNCOMPRESSED' | 'GZIP' — The compression algorithm used to compress data. - object - `format_type` 'Parquet', required - `compression_codec` 'UNCOMPRESSED' | 'SNAPPY' | 'GZIP' | 'ZSTD' — The compression algorithm used to compress data. - `partitioning` 'NO PARTITIONING' | 'DATE' | 'YEAR' | 'MONTH' | 'DAY' | 'YEAR/MONTH' | 'YEAR/MONTH/DAY' — Partition data by cursor fields when a cursor field is a date - `glue_catalog_float_as_decimal` boolean — Cast float/double as decimal(38,18). This can help achieve higher accuracy and represent numbers correctly as received from the source. - `destinationType` 'aws-datalake', required - DestinationAzureBlobStorage - `azure_blob_storage_endpoint_domain_name` string — This is Azure Blob Storage endpoint domain name. Leave default value (or leave it empty if run container from command line) to use Microsoft native from example. - `azure_blob_storage_container_name` string — The name of the Azure blob storage container. If not exists - will be created automatically. May be empty, then will be created automatically airbytecontainer+timestamp - `azure_blob_storage_account_name` string, required — The account's name of the Azure Blob Storage. - `azure_blob_storage_account_key` string, required — The Azure blob storage account key. - `azure_blob_storage_output_buffer_size` integer — The amount of megabytes to buffer for the output stream to Azure. This will impact memory footprint on workers, but may need adjustment for performance and appropriate block size in Azure. - `azure_blob_storage_spill_size` integer — The amount of megabytes after which the connector should spill the records in a new blob object. Make sure to configure size greater than individual records. Enter 0 if not applicable - `format` union, required — Output data format - object - `format_type` 'CSV', required - `flattening` 'No flattening' | 'Root level flattening', required — Whether the input json data should be normalized (flattened) in the output CSV. Please refer to docs for details. - object - `format_type` 'JSONL', required - `destinationType` 'azure-blob-storage', required - DestinationBigquery - `project_id` string, required — The GCP project ID for the project containing the target BigQuery dataset. Read more here. - `dataset_location` 'US' | 'EU' | 'asia-east1' | 'asia-east2' | 'asia-northeast1' | 'asia-northeast2' | 'asia-northeast3' | 'asia-south1' | 'asia-south2' | 'asia-southeast1' | 'asia-southeast2' | 'australia-southeast1' | 'australia-southeast2' | 'europe-central1' | 'europe-central2' | 'europe-north1' | 'europe-southwest1' | 'europe-west1' | 'europe-west2' | 'europe-west3' | 'europe-west4' | 'europe-west6' | 'europe-west7' | 'europe-west8' | 'europe-west9' | 'europe-west12' | 'me-central1' | 'me-central2' | 'me-west1' | 'northamerica-northeast1' | 'northamerica-northeast2' | 'southamerica-east1' | 'southamerica-west1' | 'us-central1' | 'us-east1' | 'us-east2' | 'us-east3' | 'us-east4' | 'us-east5' | 'us-south1' | 'us-west1' | 'us-west2' | 'us-west3' | 'us-west4', required — The location of the dataset. Warning: Changes made after creation will not be applied. Read more here. - `dataset_id` string, required — The default BigQuery Dataset ID that tables are replicated to if the source does not specify a namespace. Read more here. - `loading_method` union — The way data will be uploaded to BigQuery. - object — (recommended) Writes large batches of records to a file, uploads the file to GCS, then uses COPY INTO to load your data into BigQuery. Provides best-in-class speed, reliability and scalability. Read more about GCS Staging here. - `method` 'GCS Staging', required - `credential` object, required — An HMAC key is a type of credential and can be associated with a service account or a user account in Cloud Storage. Read more here. - `credential_type` 'HMAC_KEY', required - `hmac_key_access_id` string, required — HMAC key access ID. When linked to a service account, this ID is 61 characters long; when linked to a user account, it is 24 characters long. - `hmac_key_secret` string, required — The corresponding secret for the access ID. It is a 40-character base-64 encoded string. - `gcs_bucket_name` string, required — The name of the GCS bucket. Read more here. - `gcs_bucket_path` string, required — Directory under the GCS bucket where data will be written. - `keep_files_in_gcs-bucket` 'Delete all tmp files from GCS' | 'Keep all tmp files in GCS' — This upload method is supposed to temporary store records in GCS bucket. By this select you can chose if these records should be removed from GCS when migration has finished. The default "Delete all tmp files from GCS" value is used if not set explicitly. - object — (not recommended) Direct loading using SQL INSERT statements. This method is extremely inefficient and provided only for quick testing. In all other cases, you should use GCS staging. - `method` 'Standard', required - `credentials_json` string — The contents of the JSON service account key. Check out the docs if you need help generating this key. Default credentials will be used if this field is left empty. - `transformation_priority` 'interactive' | 'batch' — Interactive run type means that the query is executed as soon as possible, and these queries count towards concurrent rate limit and daily limit. Read more about interactive run type here. Batch queries are queued and started as soon as idle resources are available in the BigQuery shared resource pool, which usually occurs within a few minutes. Batch queries don’t count towards your concurrent rate limit. Read more about batch queries here. The default "interactive" value is used if not set explicitly. - `big_query_client_buffer_size_mb` integer — Google BigQuery client's chunk (buffer) size (MIN=1, MAX = 15) for each table. The size that will be written by a single RPC. Written data will be buffered and only flushed upon reaching this size or closing the channel. The default 15MB value is used if not set explicitly. Read more here. - `raw_data_dataset` string — The dataset to write raw tables into (default: airbyte_internal) - `disable_type_dedupe` boolean — Disable Writing Final Tables. WARNING! The data format in _airbyte_data is likely stable but there are no guarantees that other metadata columns will remain the same in future versions - `destinationType` 'bigquery', required - DestinationClickhouse - `host` string, required — Hostname of the database. - `port` integer, required — HTTP port of the database. - `database` string, required — Name of the database. - `username` string, required — Username to use to access the database. - `password` string — Password associated with the username. - `jdbc_url_params` string — Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (example: key1=value1&key2=value2&key3=value3). - `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use. - object - `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database - object - `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host. - `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa ) - object - `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host - `tunnel_user_password` string, required — OS-level password for logging into the jump server host - `destinationType` 'clickhouse', required - DestinationConvex - `deployment_url` string, required — URL of the Convex deployment that is the destination - `access_key` string, required — API access key used to send data to a Convex deployment. - `destinationType` 'convex', required - DestinationCumulio - `api_host` string, required — URL of the Cumul.io API (e.g. 'https://api.cumul.io', 'https://api.us.cumul.io', or VPC-specific API url). Defaults to 'https://api.cumul.io'. - `api_key` string, required — An API key generated in Cumul.io's platform (can be generated here: https://app.cumul.io/start/profile/integration). - `api_token` string, required — The corresponding API token generated in Cumul.io's platform (can be generated here: https://app.cumul.io/start/profile/integration). - `destinationType` 'cumulio', required - DestinationDatabend - `host` string, required — Hostname of the database. - `port` integer — Port of the database. - `database` string, required — Name of the database. - `table` string — The default table was written to. - `username` string, required — Username to use to access the database. - `password` string — Password associated with the username. - `destinationType` 'databend', required - DestinationDatabricks - `accept_terms` boolean, required — You must agree to the Databricks JDBC Driver Terms & Conditions to use this connector. - `databricks_server_hostname` string, required — Databricks Cluster Server Hostname. - `databricks_http_path` string, required — Databricks Cluster HTTP Path. - `databricks_port` string — Databricks Cluster Port. - `databricks_personal_access_token` string, required — Databricks Personal Access Token for making authenticated requests. - `database` string — The name of the catalog. If not specified otherwise, the "hive_metastore" will be used. - `schema` string — The default schema tables are written. If not specified otherwise, the "default" will be used. - `enable_schema_evolution` boolean — Support schema evolution for all streams. If "false", the connector might fail when a stream's schema changes. - `data_source` union, required — Storage on which the delta lake is built. - object - `data_source_type` 'MANAGED_TABLES_STORAGE', required - object - `data_source_type` 'S3_STORAGE', required - `s3_bucket_name` string, required — The name of the S3 bucket to use for intermittent staging of the data. - `s3_bucket_path` string, required — The directory under the S3 bucket where data will be written. - `s3_bucket_region` '' | 'us-east-1' | 'us-east-2' | 'us-west-1' | 'us-west-2' | 'af-south-1' | 'ap-east-1' | 'ap-south-1' | 'ap-northeast-1' | 'ap-northeast-2' | 'ap-northeast-3' | 'ap-southeast-1' | 'ap-southeast-2' | 'ca-central-1' | 'cn-north-1' | 'cn-northwest-1' | 'eu-central-1' | 'eu-north-1' | 'eu-south-1' | 'eu-west-1' | 'eu-west-2' | 'eu-west-3' | 'sa-east-1' | 'me-south-1' | 'us-gov-east-1' | 'us-gov-west-1', required — The region of the S3 staging bucket to use if utilising a copy strategy. - `s3_access_key_id` string, required — The Access Key Id granting allow one to access the above S3 staging bucket. Airbyte requires Read and Write permissions to the given bucket. - `s3_secret_access_key` string, required — The corresponding secret to the above access key id. - `file_name_pattern` string — The pattern allows you to set the file-name format for the S3 staging file(s) - object - `data_source_type` 'AZURE_BLOB_STORAGE', required - `azure_blob_storage_endpoint_domain_name` string — This is Azure Blob Storage endpoint domain name. Leave default value (or leave it empty if run container from command line) to use Microsoft native from example. - `azure_blob_storage_account_name` string, required — The account's name of the Azure Blob Storage. - `azure_blob_storage_container_name` string, required — The name of the Azure blob storage container. - `azure_blob_storage_sas_token` string, required — Shared access signature (SAS) token to grant limited access to objects in your storage account. - `purge_staging_data` boolean — Default to 'true'. Switch it to 'false' for debugging purpose. - `destinationType` 'databricks', required - DestinationDevNull - `test_destination` object, required — The type of destination to be used - `test_destination_type` 'SILENT', required - `destinationType` 'dev-null', required - DestinationDuckdb - `motherduck_api_key` string — API key to use for authentication to a MotherDuck database. - `destination_path` string, required — Path to the .duckdb file, or the text 'md:' to connect to MotherDuck. The file will be placed inside that local mount. For more information check out our docs - `schema` string — Database schema name, default for duckdb is 'main'. - `destinationType` 'duckdb', required - DestinationDynamodb - `dynamodb_endpoint` string — This is your DynamoDB endpoint url.(if you are working with AWS DynamoDB, just leave empty). - `dynamodb_table_name_prefix` string, required — The prefix to use when naming DynamoDB tables. - `dynamodb_region` '' | 'af-south-1' | 'ap-east-1' | 'ap-northeast-1' | 'ap-northeast-2' | 'ap-northeast-3' | 'ap-south-1' | 'ap-south-2' | 'ap-southeast-1' | 'ap-southeast-2' | 'ap-southeast-3' | 'ap-southeast-4' | 'ca-central-1' | 'ca-west-1' | 'cn-north-1' | 'cn-northwest-1' | 'eu-central-1' | 'eu-central-2' | 'eu-north-1' | 'eu-south-1' | 'eu-south-2' | 'eu-west-1' | 'eu-west-2' | 'eu-west-3' | 'il-central-1' | 'me-central-1' | 'me-south-1' | 'sa-east-1' | 'us-east-1' | 'us-east-2' | 'us-gov-east-1' | 'us-gov-west-1' | 'us-west-1' | 'us-west-2', required — The region of the DynamoDB. - `access_key_id` string, required — The access key id to access the DynamoDB. Airbyte requires Read and Write permissions to the DynamoDB. - `secret_access_key` string, required — The corresponding secret to the access key id. - `destinationType` 'dynamodb', required - DestinationElasticsearch - `endpoint` string, required — The full url of the Elasticsearch server - `upsert` boolean — If a primary key identifier is defined in the source, an upsert will be performed using the primary key value as the elasticsearch doc id. Does not support composite primary keys. - `ca_certificate` string — CA certificate - `authenticationMethod` union — The type of authentication to be used - object — Use a api key and secret combination to authenticate - `method` 'secret', required - `apiKeyId` string, required — The Key ID to used when accessing an enterprise Elasticsearch instance. - `apiKeySecret` string, required — The secret associated with the API Key ID. - object — Basic auth header with a username and password - `method` 'basic', required - `username` string, required — Basic auth username to access a secure Elasticsearch server - `password` string, required — Basic auth password to access a secure Elasticsearch server - `destinationType` 'elasticsearch', required - DestinationFirebolt - `username` string, required — Firebolt email address you use to login. - `password` string, required — Firebolt password. - `account` string — Firebolt account to login. - `host` string — The host name of your Firebolt database. - `database` string, required — The database to connect to. - `engine` string — Engine name or url to connect to. - `loading_method` union — Loading method used to select the way data will be uploaded to Firebolt - object - `method` 'SQL', required - object - `method` 'S3', required - `s3_bucket` string, required — The name of the S3 bucket. - `s3_region` string, required — Region name of the S3 bucket. - `aws_key_id` string, required — AWS access key granting read and write access to S3. - `aws_key_secret` string, required — Corresponding secret part of the AWS Key - `destinationType` 'firebolt', required - DestinationFirestore - `project_id` string, required — The GCP project ID for the project containing the target BigQuery dataset. - `credentials_json` string — The contents of the JSON service account key. Check out the docs if you need help generating this key. Default credentials will be used if this field is left empty. - `destinationType` 'firestore', required - DestinationGcs - `gcs_bucket_name` string, required — You can find the bucket name in the App Engine Admin console Application Settings page, under the label Google Cloud Storage Bucket. Read more here. - `gcs_bucket_path` string, required — GCS Bucket Path string Subdirectory under the above bucket to sync the data into. - `gcs_bucket_region` 'northamerica-northeast1' | 'northamerica-northeast2' | 'us-central1' | 'us-east1' | 'us-east4' | 'us-west1' | 'us-west2' | 'us-west3' | 'us-west4' | 'southamerica-east1' | 'southamerica-west1' | 'europe-central2' | 'europe-north1' | 'europe-west1' | 'europe-west2' | 'europe-west3' | 'europe-west4' | 'europe-west6' | 'asia-east1' | 'asia-east2' | 'asia-northeast1' | 'asia-northeast2' | 'asia-northeast3' | 'asia-south1' | 'asia-south2' | 'asia-southeast1' | 'asia-southeast2' | 'australia-southeast1' | 'australia-southeast2' | 'asia' | 'eu' | 'us' | 'asia1' | 'eur4' | 'nam4' — Select a Region of the GCS Bucket. Read more here. - `credential` object, required — An HMAC key is a type of credential and can be associated with a service account or a user account in Cloud Storage. Read more here. - `credential_type` 'HMAC_KEY', required - `hmac_key_access_id` string, required — When linked to a service account, this ID is 61 characters long; when linked to a user account, it is 24 characters long. Read more here. - `hmac_key_secret` string, required — The corresponding secret for the access ID. It is a 40-character base-64 encoded string. Read more here. - `format` union, required — Output data format. One of the following formats must be selected - AVRO format, PARQUET format, CSV format, or JSONL format. - object - `format_type` 'Avro', required - `compression_codec` union, required — The compression algorithm used to compress data. Default to no compression. - object - … - object - … - object - … - object - … - object - … - object - … - object - `format_type` 'CSV', required - `flattening` 'No flattening' | 'Root level flattening' — Whether the input JSON data should be normalized (flattened) in the output CSV. Please refer to docs for details. - `compression` union — Whether the output files should be compressed. If compression is selected, the output filename will have an extra extension (GZIP: ".csv.gz"). - object - … - object - … - object - `format_type` 'JSONL', required - `compression` union — Whether the output files should be compressed. If compression is selected, the output filename will have an extra extension (GZIP: ".jsonl.gz"). - object - … - object - … - object - `format_type` 'Parquet', required - `compression_codec` 'UNCOMPRESSED' | 'SNAPPY' | 'GZIP' | 'LZO' | 'BROTLI' | 'LZ4' | 'ZSTD' — The compression algorithm used to compress data pages. - `block_size_mb` integer — This is the size of a row group being buffered in memory. It limits the memory usage when writing. Larger values will improve the IO when reading, but consume more memory when writing. Default: 128 MB. - `max_padding_size_mb` integer — Maximum size allowed as padding to align row groups. This is also the minimum size of a row group. Default: 8 MB. - `page_size_kb` integer — The page size is for compression. A block is composed of pages. A page is the smallest unit that must be read fully to access a single record. If this value is too small, the compression will deteriorate. Default: 1024 KB. - `dictionary_page_size_kb` integer — There is one dictionary page per column per row group when dictionary encoding is used. The dictionary page size works like the page size but for dictionary. Default: 1024 KB. - `dictionary_encoding` boolean — Default: true. - `destinationType` 'gcs', required - DestinationKeen - `project_id` string, required — To get Keen Project ID, navigate to the Access tab from the left-hand, side panel and check the Project Details section. - `api_key` string, required — To get Keen Master API Key, navigate to the Access tab from the left-hand, side panel and check the Project Details section. - `infer_timestamp` boolean — Allow connector to guess keen.timestamp value based on the streamed data. - `destinationType` 'keen', required - DestinationKinesis - `endpoint` string, required — AWS Kinesis endpoint. - `region` string, required — AWS region. Your account determines the Regions that are available to you. - `shardCount` integer, required — Number of shards to which the data should be streamed. - `accessKey` string, required — Generate the AWS Access Key for current user. - `privateKey` string, required — The AWS Private Key - a string of numbers and letters that are unique for each account, also known as a "recovery phrase". - `bufferSize` integer, required — Buffer size for storing kinesis records before being batch streamed. - `destinationType` 'kinesis', required - DestinationLangchain - `processing` object, required - `chunk_size` integer, required — Size of chunks in tokens to store in vector store (make sure it is not too big for the context if your LLM) - `chunk_overlap` integer — Size of overlap between chunks in tokens to store in vector store to better capture relevant context - `text_fields` string[], required — List of fields in the record that should be used to calculate the embedding. All other fields are passed along as meta fields. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered text fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. - `embedding` union, required — Embedding configuration - object — Use the OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `mode` 'openai' - `openai_key` string, required - object — Use a fake embedding made out of random vectors with 1536 embedding dimensions. This is useful for testing the data pipeline without incurring any costs. - `mode` 'fake' - `indexing` union, required — Indexing configuration - object — Pinecone is a popular vector store that can be used to store and retrieve embeddings. It is a managed service and can also be queried from outside of langchain. - `mode` 'pinecone' - `pinecone_key` string, required - `pinecone_environment` string, required — Pinecone environment to use - `index` string, required — Pinecone index to use - object — DocArrayHnswSearch is a lightweight Document Index implementation provided by Docarray that runs fully locally and is best suited for small- to medium-sized datasets. It stores vectors on disk in hnswlib, and stores all other data in SQLite. - `mode` 'DocArrayHnswSearch' - `destination_path` string, required — Path to the directory where hnswlib and meta data files will be written. The files will be placed inside that local mount. All files in the specified destination directory will be deleted on each run. - object — Chroma is a popular vector store that can be used to store and retrieve embeddings. It will build its index in memory and persist it to disk by the end of the sync. - `mode` 'chroma_local' - `destination_path` string, required — Path to the directory where chroma files will be written. The files will be placed inside that local mount. - `collection_name` string — Name of the collection to use. - `destinationType` 'langchain', required - DestinationMilvus — The configuration model for the Vector DB based destinations. This model is used to generate the UI for the destination configuration, as well as to provide type safety for the configuration passed to the destination. The configuration model is composed of four parts: * Processing configuration * Embedding configuration * Indexing configuration * Advanced configuration Processing, embedding and advanced configuration are provided by this base class, while the indexing configuration is provided by the destination connector in the sub class. - `embedding` union, required — Embedding configuration - object — Use the OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `mode` 'openai', required - `openai_key` string, required - object — Use the Cohere API to embed text. - `mode` 'cohere', required - `cohere_key` string, required - object — Use a fake embedding made out of random vectors with 1536 embedding dimensions. This is useful for testing the data pipeline without incurring any costs. - `mode` 'fake', required - object — Use the Azure-hosted OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `mode` 'azure_openai', required - `openai_key` string, required — The API key for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `api_base` string, required — The base URL for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `deployment` string, required — The deployment for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - object — Use a service that's compatible with the OpenAI API to embed text. - `mode` 'openai_compatible', required - `api_key` string - `base_url` string, required — The base URL for your OpenAI-compatible service - `model_name` string — The name of the model to use for embedding - `dimensions` integer, required — The number of dimensions the embedding model is generating - `processing` object, required - `chunk_size` integer, required — Size of chunks in tokens to store in vector store (make sure it is not too big for the context if your LLM) - `chunk_overlap` integer — Size of overlap between chunks in tokens to store in vector store to better capture relevant context - `text_fields` string[] — List of fields in the record that should be used to calculate the embedding. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered text fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. - `metadata_fields` string[] — List of fields in the record that should be stored as metadata. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered metadata fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. When specifying nested paths, all matching values are flattened into an array set to a field named by the path. - `text_splitter` union — Split text fields into chunks based on the specified method. - object — Split the text by the list of separators until the chunk size is reached, using the earlier mentioned separators where possible. This is useful for splitting text fields by paragraphs, sentences, words, etc. - `mode` 'separator', required - `separators` string[] — List of separator strings to split text fields by. The separator itself needs to be wrapped in double quotes, e.g. to split by the dot character, use ".". To split by a newline, use "\n". - `keep_separator` boolean — Whether to keep the separator in the resulting chunks - object — Split the text by Markdown headers down to the specified header level. If the chunk size fits multiple sections, they will be combined into a single chunk. - `mode` 'markdown', required - `split_level` integer — Level of markdown headers to split text fields by. Headings down to the specified level will be used as split points - object — Split the text by suitable delimiters based on the programming language. This is useful for splitting code into chunks. - `mode` 'code', required - `language` 'cpp' | 'go' | 'java' | 'js' | 'php' | 'proto' | 'python' | 'rst' | 'ruby' | 'rust' | 'scala' | 'swift' | 'markdown' | 'latex' | 'html' | 'sol', required — Split code in suitable places based on the programming language - `field_name_mappings` object[] — List of fields to rename. Not applicable for nested fields, but can be used to rename fields already flattened via dot notation. - `from_field` string, required — The field name in the source - `to_field` string, required — The field name to use in the destination - `omit_raw_text` boolean — Do not store the text that gets embedded along with the vector and the metadata in the destination. If set to true, only the vector and the metadata will be stored - in this case raw text for LLM use cases needs to be retrieved from another source. - `indexing` object, required — Indexing configuration - `host` string, required — The public endpoint of the Milvus instance. - `db` string — The database to connect to - `collection` string, required — The collection to load data into - `auth` union, required — Authentication method - object — Authenticate using an API token (suitable for Zilliz Cloud) - `mode` 'token', required - `token` string, required — API Token for the Milvus instance - object — Authenticate using username and password (suitable for self-managed Milvus clusters) - `mode` 'username_password', required - `username` string, required — Username for the Milvus instance - `password` string, required — Password for the Milvus instance - object — Do not authenticate (suitable for locally running test clusters, do not use for clusters with public IP addresses) - `mode` 'no_auth', required - `vector_field` string — The field in the entity that contains the vector - `text_field` string — The field in the entity that contains the embedded text - `destinationType` 'milvus', required - DestinationMongodb - `instance_type` union — MongoDb instance to connect to. For MongoDB Atlas and Replica Set TLS connection is used by default. - object - `instance` 'standalone', required - `host` string, required — The Host of a Mongo database to be replicated. - `port` integer, required — The Port of a Mongo database to be replicated. - object - `instance` 'replica', required - `server_addresses` string, required — The members of a replica set. Please specify `host`:`port` of each member seperated by comma. - `replica_set` string — A replica set name. - object - `instance` 'atlas', required - `cluster_url` string, required — URL of a cluster to connect to. - `database` string, required — Name of the database. - `auth_type` union, required — Authorization type. - object — None. - `authorization` 'none', required - object — Login/Password. - `authorization` 'login/password', required - `username` string, required — Username to use to access the database. - `password` string, required — Password associated with the username. - `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use. - object - `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database - object - `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host. - `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa ) - object - `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host - `tunnel_user_password` string, required — OS-level password for logging into the jump server host - `destinationType` 'mongodb', required - DestinationMssql - `host` string, required — The host name of the MSSQL database. - `port` integer, required — The port of the MSSQL database. - `database` string, required — The name of the MSSQL database. - `schema` string, required — The default schema tables are written to if the source does not specify a namespace. The usual value for this field is "public". - `username` string, required — The username which is used to access the database. - `password` string — The password associated with this username. - `jdbc_url_params` string — Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (example: key1=value1&key2=value2&key3=value3). - `ssl_method` union — The encryption method which is used to communicate with the database. - object — Use the certificate provided by the server without verification. (For testing purposes only!) - `ssl_method` 'encrypted_trust_server_certificate', required - object — Verify and use the certificate provided by the server. - `ssl_method` 'encrypted_verify_certificate', required - `hostNameInCertificate` string — Specifies the host name of the server. The value of this property must match the subject property of the certificate. - `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use. - object - `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database - object - `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host. - `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa ) - object - `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host - `tunnel_user_password` string, required — OS-level password for logging into the jump server host - `destinationType` 'mssql', required - DestinationMysql - `host` string, required — Hostname of the database. - `port` integer, required — Port of the database. - `database` string, required — Name of the database. - `username` string, required — Username to use to access the database. - `password` string — Password associated with the username. - `jdbc_url_params` string — Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (example: key1=value1&key2=value2&key3=value3). - `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use. - object - `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database - object - `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host. - `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa ) - object - `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host - `tunnel_user_password` string, required — OS-level password for logging into the jump server host - `destinationType` 'mysql', required - DestinationOracle - `host` string, required — The hostname of the database. - `port` integer, required — The port of the database. - `sid` string, required — The System Identifier uniquely distinguishes the instance from any other instance on the same computer. - `username` string, required — The username to access the database. This user must have CREATE USER privileges in the database. - `password` string — The password associated with the username. - `jdbc_url_params` string — Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (example: key1=value1&key2=value2&key3=value3). - `schema` string — The default schema is used as the target schema for all statements issued from the connection that do not explicitly specify a schema name. The usual value for this field is "airbyte". In Oracle, schemas and users are the same thing, so the "user" parameter is used as the login credentials and this is used for the default Airbyte message schema. - `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use. - object - `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database - object - `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host. - `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa ) - object - `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host - `tunnel_user_password` string, required — OS-level password for logging into the jump server host - `destinationType` 'oracle', required - DestinationPinecone — The configuration model for the Vector DB based destinations. This model is used to generate the UI for the destination configuration, as well as to provide type safety for the configuration passed to the destination. The configuration model is composed of four parts: * Processing configuration * Embedding configuration * Indexing configuration * Advanced configuration Processing, embedding and advanced configuration are provided by this base class, while the indexing configuration is provided by the destination connector in the sub class. - `embedding` union, required — Embedding configuration - object — Use the OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `mode` 'openai', required - `openai_key` string, required - object — Use the Cohere API to embed text. - `mode` 'cohere', required - `cohere_key` string, required - object — Use a fake embedding made out of random vectors with 1536 embedding dimensions. This is useful for testing the data pipeline without incurring any costs. - `mode` 'fake', required - object — Use the Azure-hosted OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `mode` 'azure_openai', required - `openai_key` string, required — The API key for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `api_base` string, required — The base URL for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `deployment` string, required — The deployment for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - object — Use a service that's compatible with the OpenAI API to embed text. - `mode` 'openai_compatible', required - `api_key` string - `base_url` string, required — The base URL for your OpenAI-compatible service - `model_name` string — The name of the model to use for embedding - `dimensions` integer, required — The number of dimensions the embedding model is generating - `processing` object, required - `chunk_size` integer, required — Size of chunks in tokens to store in vector store (make sure it is not too big for the context if your LLM) - `chunk_overlap` integer — Size of overlap between chunks in tokens to store in vector store to better capture relevant context - `text_fields` string[] — List of fields in the record that should be used to calculate the embedding. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered text fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. - `metadata_fields` string[] — List of fields in the record that should be stored as metadata. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered metadata fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. When specifying nested paths, all matching values are flattened into an array set to a field named by the path. - `text_splitter` union — Split text fields into chunks based on the specified method. - object — Split the text by the list of separators until the chunk size is reached, using the earlier mentioned separators where possible. This is useful for splitting text fields by paragraphs, sentences, words, etc. - `mode` 'separator', required - `separators` string[] — List of separator strings to split text fields by. The separator itself needs to be wrapped in double quotes, e.g. to split by the dot character, use ".". To split by a newline, use "\n". - `keep_separator` boolean — Whether to keep the separator in the resulting chunks - object — Split the text by Markdown headers down to the specified header level. If the chunk size fits multiple sections, they will be combined into a single chunk. - `mode` 'markdown', required - `split_level` integer — Level of markdown headers to split text fields by. Headings down to the specified level will be used as split points - object — Split the text by suitable delimiters based on the programming language. This is useful for splitting code into chunks. - `mode` 'code', required - `language` 'cpp' | 'go' | 'java' | 'js' | 'php' | 'proto' | 'python' | 'rst' | 'ruby' | 'rust' | 'scala' | 'swift' | 'markdown' | 'latex' | 'html' | 'sol', required — Split code in suitable places based on the programming language - `field_name_mappings` object[] — List of fields to rename. Not applicable for nested fields, but can be used to rename fields already flattened via dot notation. - `from_field` string, required — The field name in the source - `to_field` string, required — The field name to use in the destination - `omit_raw_text` boolean — Do not store the text that gets embedded along with the vector and the metadata in the destination. If set to true, only the vector and the metadata will be stored - in this case raw text for LLM use cases needs to be retrieved from another source. - `indexing` object, required — Pinecone is a popular vector store that can be used to store and retrieve embeddings. - `pinecone_key` string, required — The Pinecone API key to use matching the environment (copy from Pinecone console) - `pinecone_environment` string, required — Pinecone Cloud environment to use - `index` string, required — Pinecone index in your project to load data into - `destinationType` 'pinecone', required - DestinationPostgres - `host` string, required — Hostname of the database. - `port` integer, required — Port of the database. - `database` string, required — Name of the database. - `schema` string, required — The default schema tables are written to if the source does not specify a namespace. The usual value for this field is "public". - `username` string, required — Username to use to access the database. - `password` string — Password associated with the username. - `ssl_mode` union — SSL connection modes. disable - Chose this mode to disable encryption of communication between Airbyte and destination database allow - Chose this mode to enable encryption only when required by the source database prefer - Chose this mode to allow unencrypted connection only if the source database does not support encryption require - Chose this mode to always require encryption. If the source database server does not support encryption, connection will fail verify-ca - Chose this mode to always require encryption and to verify that the source database server has a valid SSL certificate verify-full - This is the most secure mode. Chose this mode to always require encryption and to verify the identity of the source database server See more information - in the docs. - object — Disable SSL. - `mode` 'disable', required - object — Allow SSL mode. - `mode` 'allow', required - object — Prefer SSL mode. - `mode` 'prefer', required - object — Require SSL mode. - `mode` 'require', required - object — Verify-ca SSL mode. - `mode` 'verify-ca', required - `ca_certificate` string, required — CA certificate - `client_key_password` string — Password for keystorage. This field is optional. If you do not add it - the password will be generated automatically. - object — Verify-full SSL mode. - `mode` 'verify-full', required - `ca_certificate` string, required — CA certificate - `client_certificate` string, required — Client certificate - `client_key` string, required — Client key - `client_key_password` string — Password for keystorage. This field is optional. If you do not add it - the password will be generated automatically. - `jdbc_url_params` string — Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (example: key1=value1&key2=value2&key3=value3). - `raw_data_schema` string — The schema to write raw tables into - `disable_type_dedupe` boolean — Disable Writing Final Tables. WARNING! The data format in _airbyte_data is likely stable but there are no guarantees that other metadata columns will remain the same in future versions - `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use. - object - `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database - object - `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host. - `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa ) - object - `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host - `tunnel_user_password` string, required — OS-level password for logging into the jump server host - `destinationType` 'postgres', required - DestinationPubsub - `project_id` string, required — The GCP project ID for the project containing the target PubSub. - `topic_id` string, required — The PubSub topic ID in the given GCP project ID. - `credentials_json` string, required — The contents of the JSON service account key. Check out the docs if you need help generating this key. - `ordering_enabled` boolean, required — If TRUE PubSub publisher will have message ordering enabled. Every message will have an ordering key of stream - `batching_enabled` boolean, required — If TRUE messages will be buffered instead of sending them one by one - `batching_delay_threshold` integer — Number of ms before the buffer is flushed - `batching_element_count_threshold` integer — Number of messages before the buffer is flushed - `batching_request_bytes_threshold` integer — Number of bytes before the buffer is flushed - `destinationType` 'pubsub', required - DestinationQdrant — The configuration model for the Vector DB based destinations. This model is used to generate the UI for the destination configuration, as well as to provide type safety for the configuration passed to the destination. The configuration model is composed of four parts: * Processing configuration * Embedding configuration * Indexing configuration * Advanced configuration Processing, embedding and advanced configuration are provided by this base class, while the indexing configuration is provided by the destination connector in the sub class. - `embedding` union, required — Embedding configuration - object — Use the OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `mode` 'openai', required - `openai_key` string, required - object — Use the Cohere API to embed text. - `mode` 'cohere', required - `cohere_key` string, required - object — Use a fake embedding made out of random vectors with 1536 embedding dimensions. This is useful for testing the data pipeline without incurring any costs. - `mode` 'fake', required - object — Use the Azure-hosted OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `mode` 'azure_openai', required - `openai_key` string, required — The API key for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `api_base` string, required — The base URL for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `deployment` string, required — The deployment for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - object — Use a service that's compatible with the OpenAI API to embed text. - `mode` 'openai_compatible', required - `api_key` string - `base_url` string, required — The base URL for your OpenAI-compatible service - `model_name` string — The name of the model to use for embedding - `dimensions` integer, required — The number of dimensions the embedding model is generating - `processing` object, required - `chunk_size` integer, required — Size of chunks in tokens to store in vector store (make sure it is not too big for the context if your LLM) - `chunk_overlap` integer — Size of overlap between chunks in tokens to store in vector store to better capture relevant context - `text_fields` string[] — List of fields in the record that should be used to calculate the embedding. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered text fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. - `metadata_fields` string[] — List of fields in the record that should be stored as metadata. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered metadata fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. When specifying nested paths, all matching values are flattened into an array set to a field named by the path. - `text_splitter` union — Split text fields into chunks based on the specified method. - object — Split the text by the list of separators until the chunk size is reached, using the earlier mentioned separators where possible. This is useful for splitting text fields by paragraphs, sentences, words, etc. - `mode` 'separator', required - `separators` string[] — List of separator strings to split text fields by. The separator itself needs to be wrapped in double quotes, e.g. to split by the dot character, use ".". To split by a newline, use "\n". - `keep_separator` boolean — Whether to keep the separator in the resulting chunks - object — Split the text by Markdown headers down to the specified header level. If the chunk size fits multiple sections, they will be combined into a single chunk. - `mode` 'markdown', required - `split_level` integer — Level of markdown headers to split text fields by. Headings down to the specified level will be used as split points - object — Split the text by suitable delimiters based on the programming language. This is useful for splitting code into chunks. - `mode` 'code', required - `language` 'cpp' | 'go' | 'java' | 'js' | 'php' | 'proto' | 'python' | 'rst' | 'ruby' | 'rust' | 'scala' | 'swift' | 'markdown' | 'latex' | 'html' | 'sol', required — Split code in suitable places based on the programming language - `field_name_mappings` object[] — List of fields to rename. Not applicable for nested fields, but can be used to rename fields already flattened via dot notation. - `from_field` string, required — The field name in the source - `to_field` string, required — The field name to use in the destination - `omit_raw_text` boolean — Do not store the text that gets embedded along with the vector and the metadata in the destination. If set to true, only the vector and the metadata will be stored - in this case raw text for LLM use cases needs to be retrieved from another source. - `indexing` object, required — Indexing configuration - `url` string, required — Public Endpoint of the Qdrant cluser - `auth_method` union — Method to authenticate with the Qdrant Instance - object - `mode` 'api_key_auth' - `api_key` string, required — API Key for the Qdrant instance - object - `mode` 'no_auth' - `prefer_grpc` boolean — Whether to prefer gRPC over HTTP. Set to true for Qdrant cloud clusters - `collection` string, required — The collection to load data into - `distance_metric` 'dot' | 'cos' | 'euc' — The Distance metric used to measure similarities among vectors. This field is only used if the collection defined in the does not exist yet and is created automatically by the connector. - `text_field` string — The field in the payload that contains the embedded text - `destinationType` 'qdrant', required - DestinationRedis - `host` string, required — Redis host to connect to. - `port` integer, required — Port of Redis. - `username` string, required — Username associated with Redis. - `password` string — Password associated with Redis. - `ssl` boolean — Indicates whether SSL encryption protocol will be used to connect to Redis. It is recommended to use SSL connection if possible. - `ssl_mode` union — SSL connection modes.
  • verify-full - This is the most secure mode. Always require encryption and verifies the identity of the source database server - object — Disable SSL. - `mode` 'disable', required - object — Verify-full SSL mode. - `mode` 'verify-full', required - `ca_certificate` string, required — CA certificate - `client_certificate` string, required — Client certificate - `client_key` string, required — Client key - `client_key_password` string — Password for keystorage. If you do not add it - the password will be generated automatically. - `cache_type` 'hash', required — Redis cache type to store data in. - `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use. - object - `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database - object - `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host. - `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa ) - object - `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host - `tunnel_user_password` string, required — OS-level password for logging into the jump server host - `destinationType` 'redis', required - DestinationRedshift - `host` string, required — Host Endpoint of the Redshift Cluster (must include the cluster-id, region and end with .redshift.amazonaws.com) - `port` integer, required — Port of the database. - `username` string, required — Username to use to access the database. - `password` string, required — Password associated with the username. - `database` string, required — Name of the database. - `schema` string, required — The default schema tables are written to if the source does not specify a namespace. Unless specifically configured, the usual value for this field is "public". - `jdbc_url_params` string — Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (example: key1=value1&key2=value2&key3=value3). - `uploading_method` union — The way data will be uploaded to Redshift. - object — (recommended) Uploads data to S3 and then uses a COPY to insert the data into Redshift. COPY is recommended for production workloads for better speed and scalability. See AWS docs for more details. - `method` 'S3 Staging', required - `s3_bucket_name` string, required — The name of the staging S3 bucket. - `s3_bucket_path` string — The directory under the S3 bucket where data will be written. If not provided, then defaults to the root directory. See path's name recommendations for more details. - `s3_bucket_region` '' | 'af-south-1' | 'ap-east-1' | 'ap-northeast-1' | 'ap-northeast-2' | 'ap-northeast-3' | 'ap-south-1' | 'ap-south-2' | 'ap-southeast-1' | 'ap-southeast-2' | 'ap-southeast-3' | 'ap-southeast-4' | 'ca-central-1' | 'ca-west-1' | 'cn-north-1' | 'cn-northwest-1' | 'eu-central-1' | 'eu-central-2' | 'eu-north-1' | 'eu-south-1' | 'eu-south-2' | 'eu-west-1' | 'eu-west-2' | 'eu-west-3' | 'il-central-1' | 'me-central-1' | 'me-south-1' | 'sa-east-1' | 'sa-east-1' | 'us-east-1' | 'us-east-2' | 'us-gov-east-1' | 'us-gov-west-1' | 'us-west-1' | 'us-west-2', required — The region of the S3 staging bucket. - `access_key_id` string, required — This ID grants access to the above S3 staging bucket. Airbyte requires Read and Write permissions to the given bucket. See AWS docs on how to generate an access key ID and secret access key. - `secret_access_key` string, required — The corresponding secret to the above access key id. See AWS docs on how to generate an access key ID and secret access key. - `file_name_pattern` string — The pattern allows you to set the file-name format for the S3 staging file(s) - `purge_staging_data` boolean — Whether to delete the staging files from S3 after completing the sync. See docs for details. - `encryption` union — How to encrypt the staging data - object — Staging data will be stored in plaintext. - … - object — Staging data will be encrypted using AES-CBC envelope encryption. - … - `file_buffer_count` integer — Number of file buffers allocated for writing data. Increasing this number is beneficial for connections using Change Data Capture (CDC) and up to the number of streams within a connection. Increasing the number of file buffers past the maximum number of streams has deteriorating effects - object — (not recommended) Direct loading using SQL INSERT statements. This method is extremely inefficient and provided only for quick testing. In all other cases, you should use S3 uploading. - `method` 'Standard', required - `raw_data_schema` string — The schema to write raw tables into - `enable_incremental_final_table_updates` boolean — When enabled your data will load into your final tables incrementally while your data is still being synced. When Disabled (the default), your data loads into your final tables once at the end of a sync. Note that this option only applies if you elect to create Final tables - `disable_type_dedupe` boolean — Disable Writing Final Tables. WARNING! The data format in _airbyte_data is likely stable but there are no guarantees that other metadata columns will remain the same in future versions - `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use. - object - `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database - object - `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host. - `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa ) - object - `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host - `tunnel_user_password` string, required — OS-level password for logging into the jump server host - `destinationType` 'redshift', required - DestinationS3 - `access_key_id` string — The access key ID to access the S3 bucket. Airbyte requires Read and Write permissions to the given bucket. Read more here. - `secret_access_key` string — The corresponding secret to the access key ID. Read more here - `s3_bucket_name` string, required — The name of the S3 bucket. Read more here. - `s3_bucket_path` string, required — Directory under the S3 bucket where data will be written. Read more here - `s3_bucket_region` '' | 'af-south-1' | 'ap-east-1' | 'ap-northeast-1' | 'ap-northeast-2' | 'ap-northeast-3' | 'ap-south-1' | 'ap-south-2' | 'ap-southeast-1' | 'ap-southeast-2' | 'ap-southeast-3' | 'ap-southeast-4' | 'ca-central-1' | 'ca-west-1' | 'cn-north-1' | 'cn-northwest-1' | 'eu-central-1' | 'eu-central-2' | 'eu-north-1' | 'eu-south-1' | 'eu-south-2' | 'eu-west-1' | 'eu-west-2' | 'eu-west-3' | 'il-central-1' | 'me-central-1' | 'me-south-1' | 'sa-east-1' | 'us-east-1' | 'us-east-2' | 'us-gov-east-1' | 'us-gov-west-1' | 'us-west-1' | 'us-west-2', required — The region of the S3 bucket. See here for all region codes. - `format` union, required — Format of the data output. See here for more details - object - `format_type` 'CSV', required - `flattening` 'No flattening' | 'Root level flattening', required — Whether the input json data should be normalized (flattened) in the output CSV. Please refer to docs for details. - `compression` union — Whether the output files should be compressed. If compression is selected, the output filename will have an extra extension (GZIP: ".csv.gz"). - object - … - object - … - object - `format_type` 'JSONL', required - `flattening` 'No flattening' | 'Root level flattening' — Whether the input json data should be normalized (flattened) in the output JSON Lines. Please refer to docs for details. - `compression` union — Whether the output files should be compressed. If compression is selected, the output filename will have an extra extension (GZIP: ".jsonl.gz"). - object - … - object - … - object - `format_type` 'Avro', required - `compression_codec` union, required — The compression algorithm used to compress data. Default to no compression. - object - … - object - … - object - … - object - … - object - … - object - … - object - `format_type` 'Parquet', required - `compression_codec` 'UNCOMPRESSED' | 'SNAPPY' | 'GZIP' | 'LZO' | 'BROTLI' | 'LZ4' | 'ZSTD' — The compression algorithm used to compress data pages. - `block_size_mb` integer — This is the size of a row group being buffered in memory. It limits the memory usage when writing. Larger values will improve the IO when reading, but consume more memory when writing. Default: 128 MB. - `max_padding_size_mb` integer — Maximum size allowed as padding to align row groups. This is also the minimum size of a row group. Default: 8 MB. - `page_size_kb` integer — The page size is for compression. A block is composed of pages. A page is the smallest unit that must be read fully to access a single record. If this value is too small, the compression will deteriorate. Default: 1024 KB. - `dictionary_page_size_kb` integer — There is one dictionary page per column per row group when dictionary encoding is used. The dictionary page size works like the page size but for dictionary. Default: 1024 KB. - `dictionary_encoding` boolean — Default: true. - `s3_endpoint` string — Your S3 endpoint url. Read more here - `s3_path_format` string — Format string on how data will be organized inside the S3 bucket directory. Read more here - `file_name_pattern` string — The pattern allows you to set the file-name format for the S3 staging file(s) - `destinationType` 's3', required - DestinationS3Glue - `access_key_id` string — The access key ID to access the S3 bucket. Airbyte requires Read and Write permissions to the given bucket. Read more here. - `secret_access_key` string — The corresponding secret to the access key ID. Read more here - `s3_bucket_name` string, required — The name of the S3 bucket. Read more here. - `s3_bucket_path` string, required — Directory under the S3 bucket where data will be written. Read more here - `s3_bucket_region` '' | 'af-south-1' | 'ap-east-1' | 'ap-northeast-1' | 'ap-northeast-2' | 'ap-northeast-3' | 'ap-south-1' | 'ap-south-2' | 'ap-southeast-1' | 'ap-southeast-2' | 'ap-southeast-3' | 'ap-southeast-4' | 'ca-central-1' | 'ca-west-1' | 'cn-north-1' | 'cn-northwest-1' | 'eu-central-1' | 'eu-central-2' | 'eu-north-1' | 'eu-south-1' | 'eu-south-2' | 'eu-west-1' | 'eu-west-2' | 'eu-west-3' | 'il-central-1' | 'me-central-1' | 'me-south-1' | 'sa-east-1' | 'us-east-1' | 'us-east-2' | 'us-gov-east-1' | 'us-gov-west-1' | 'us-west-1' | 'us-west-2', required — The region of the S3 bucket. See here for all region codes. - `format` object, required — Format of the data output. See here for more details - `format_type` 'JSONL', required - `compression` union — Whether the output files should be compressed. If compression is selected, the output filename will have an extra extension (GZIP: ".jsonl.gz"). - object - `compression_type` 'No Compression' - object - `compression_type` 'GZIP' - `flattening` 'No flattening' | 'Root level flattening' — Whether the input json data should be normalized (flattened) in the output JSON Lines. Please refer to docs for details. - `s3_endpoint` string — Your S3 endpoint url. Read more here - `s3_path_format` string — Format string on how data will be organized inside the S3 bucket directory. Read more here - `file_name_pattern` string — The pattern allows you to set the file-name format for the S3 staging file(s) - `glue_database` string, required — Name of the glue database for creating the tables, leave blank if no integration - `glue_serialization_library` 'org.openx.data.jsonserde.JsonSerDe' | 'org.apache.hive.hcatalog.data.JsonSerDe', required — The library that your query engine will use for reading and writing data in your lake. - `destinationType` 's3-glue', required - DestinationSftpJson - `host` string, required — Hostname of the SFTP server. - `port` integer — Port of the SFTP server. - `username` string, required — Username to use to access the SFTP server. - `password` string, required — Password associated with the username. - `destination_path` string, required — Path to the directory where json files will be written. - `destinationType` 'sftp-json', required - DestinationSnowflake - `host` string, required — Enter your Snowflake account's locator (in the format ...snowflakecomputing.com) - `role` string, required — Enter the role that you want to use to access Snowflake - `warehouse` string, required — Enter the name of the warehouse that you want to sync data into - `database` string, required — Enter the name of the database you want to sync data into - `schema` string, required — Enter the name of the default schema - `username` string, required — Enter the name of the user you want to use to access the database - `credentials` union - object - `auth_type` 'Key Pair Authentication' - `private_key` string, required — RSA Private key to use for Snowflake connection. See the docs for more information on how to obtain this key. - `private_key_password` string — Passphrase for private key - object - `auth_type` 'Username and Password' - `password` string, required — Enter the password associated with the username. - object - `auth_type` 'OAuth2.0' - `client_id` string — Enter your application's Client ID - `client_secret` string — Enter your application's Client secret - `access_token` string, required — Enter you application's Access Token - `refresh_token` string, required — Enter your application's Refresh Token - `jdbc_url_params` string — Enter the additional properties to pass to the JDBC URL string when connecting to the database (formatted as key=value pairs separated by the symbol &). Example: key1=value1&key2=value2&key3=value3 - `raw_data_schema` string — The schema to write raw tables into (default: airbyte_internal) - `disable_type_dedupe` boolean — Disable Writing Final Tables. WARNING! The data format in _airbyte_data is likely stable but there are no guarantees that other metadata columns will remain the same in future versions - `enable_incremental_final_table_updates` boolean — When enabled your data will load into your final tables incrementally while your data is still being synced. When Disabled (the default), your data loads into your final tables once at the end of a sync. Note that this option only applies if you elect to create Final tables - `destinationType` 'snowflake', required - DestinationTeradata - `host` string, required — Hostname of the database. - `username` string, required — Username to use to access the database. - `password` string — Password associated with the username. - `schema` string — The default schema tables are written to if the source does not specify a namespace. The usual value for this field is "public". - `ssl` boolean — Encrypt data using SSL. When activating SSL, please select one of the connection modes. - `ssl_mode` union — SSL connection modes. disable - Chose this mode to disable encryption of communication between Airbyte and destination database allow - Chose this mode to enable encryption only when required by the destination database prefer - Chose this mode to allow unencrypted connection only if the destination database does not support encryption require - Chose this mode to always require encryption. If the destination database server does not support encryption, connection will fail verify-ca - Chose this mode to always require encryption and to verify that the destination database server has a valid SSL certificate verify-full - This is the most secure mode. Chose this mode to always require encryption and to verify the identity of the destination database server See more information - in the docs. - object — Disable SSL. - `mode` 'disable', required - object — Allow SSL mode. - `mode` 'allow', required - object — Prefer SSL mode. - `mode` 'prefer', required - object — Require SSL mode. - `mode` 'require', required - object — Verify-ca SSL mode. - `mode` 'verify-ca', required - `ssl_ca_certificate` string, required — Specifies the file name of a PEM file that contains Certificate Authority (CA) certificates for use with SSLMODE=verify-ca. See more information - in the docs. - object — Verify-full SSL mode. - `mode` 'verify-full', required - `ssl_ca_certificate` string, required — Specifies the file name of a PEM file that contains Certificate Authority (CA) certificates for use with SSLMODE=verify-full. See more information - in the docs. - `jdbc_url_params` string — Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (example: key1=value1&key2=value2&key3=value3). - `destinationType` 'teradata', required - DestinationTimeplus - `endpoint` string, required — Timeplus workspace endpoint - `apikey` string, required — Personal API key - `destinationType` 'timeplus', required - DestinationTypesense - `api_key` string, required — Typesense API Key - `host` string, required — Hostname of the Typesense instance without protocol. - `port` string — Port of the Typesense instance. Ex: 8108, 80, 443. Default is 443 - `protocol` string — Protocol of the Typesense instance. Ex: http or https. Default is https - `batch_size` integer — How many documents should be imported together. Default 1000 - `destinationType` 'typesense', required - DestinationVectara — Configuration to connect to the Vectara instance - `oauth2` object, required — OAuth2.0 credentials used to authenticate admin actions (creating/deleting corpora) - `client_id` string, required — OAuth2.0 client id - `client_secret` string, required — OAuth2.0 client secret - `customer_id` string, required — Your customer id as it is in the authenticaion url - `corpus_name` string, required — The Name of Corpus to load data into - `parallelize` boolean — Parallelize indexing into Vectara with multiple threads - `text_fields` string[] — List of fields in the record that should be in the section of the document. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered text fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. - `title_field` string — A field that will be used to populate the `title` of each document. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered text fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. - `metadata_fields` string[] — List of fields in the record that should be stored as metadata. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered metadata fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. When specifying nested paths, all matching values are flattened into an array set to a field named by the path. - `destinationType` 'vectara', required - DestinationVertica - `host` string, required — Hostname of the database. - `port` integer, required — Port of the database. - `database` string, required — Name of the database. - `username` string, required — Username to use to access the database. - `password` string — Password associated with the username. - `jdbc_url_params` string — Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (example: key1=value1&key2=value2&key3=value3). - `schema` string, required — Schema for vertica destination - `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use. - object - `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database - object - `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host. - `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa ) - object - `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host - `tunnel_user_password` string, required — OS-level password for logging into the jump server host - `destinationType` 'vertica', required - DestinationWeaviate — The configuration model for the Vector DB based destinations. This model is used to generate the UI for the destination configuration, as well as to provide type safety for the configuration passed to the destination. The configuration model is composed of four parts: * Processing configuration * Embedding configuration * Indexing configuration * Advanced configuration Processing, embedding and advanced configuration are provided by this base class, while the indexing configuration is provided by the destination connector in the sub class. - `embedding` union, required — Embedding configuration - object — Do not calculate and pass embeddings to Weaviate. Suitable for clusters with configured vectorizers to calculate embeddings within Weaviate or for classes that should only support regular text search. - `mode` 'no_embedding', required - object — Use the Azure-hosted OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `mode` 'azure_openai', required - `openai_key` string, required — The API key for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `api_base` string, required — The base URL for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `deployment` string, required — The deployment for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - object — Use the OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `mode` 'openai', required - `openai_key` string, required - object — Use the Cohere API to embed text. - `mode` 'cohere', required - `cohere_key` string, required - object — Use a field in the record as the embedding. This is useful if you already have an embedding for your data and want to store it in the vector store. - `mode` 'from_field', required - `field_name` string, required — Name of the field in the record that contains the embedding - `dimensions` integer, required — The number of dimensions the embedding model is generating - object — Use a fake embedding made out of random vectors with 1536 embedding dimensions. This is useful for testing the data pipeline without incurring any costs. - `mode` 'fake', required - object — Use a service that's compatible with the OpenAI API to embed text. - `mode` 'openai_compatible', required - `api_key` string - `base_url` string, required — The base URL for your OpenAI-compatible service - `model_name` string — The name of the model to use for embedding - `dimensions` integer, required — The number of dimensions the embedding model is generating - `processing` object, required - `chunk_size` integer, required — Size of chunks in tokens to store in vector store (make sure it is not too big for the context if your LLM) - `chunk_overlap` integer — Size of overlap between chunks in tokens to store in vector store to better capture relevant context - `text_fields` string[] — List of fields in the record that should be used to calculate the embedding. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered text fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. - `metadata_fields` string[] — List of fields in the record that should be stored as metadata. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered metadata fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. When specifying nested paths, all matching values are flattened into an array set to a field named by the path. - `text_splitter` union — Split text fields into chunks based on the specified method. - object — Split the text by the list of separators until the chunk size is reached, using the earlier mentioned separators where possible. This is useful for splitting text fields by paragraphs, sentences, words, etc. - `mode` 'separator', required - `separators` string[] — List of separator strings to split text fields by. The separator itself needs to be wrapped in double quotes, e.g. to split by the dot character, use ".". To split by a newline, use "\n". - `keep_separator` boolean — Whether to keep the separator in the resulting chunks - object — Split the text by Markdown headers down to the specified header level. If the chunk size fits multiple sections, they will be combined into a single chunk. - `mode` 'markdown', required - `split_level` integer — Level of markdown headers to split text fields by. Headings down to the specified level will be used as split points - object — Split the text by suitable delimiters based on the programming language. This is useful for splitting code into chunks. - `mode` 'code', required - `language` 'cpp' | 'go' | 'java' | 'js' | 'php' | 'proto' | 'python' | 'rst' | 'ruby' | 'rust' | 'scala' | 'swift' | 'markdown' | 'latex' | 'html' | 'sol', required — Split code in suitable places based on the programming language - `field_name_mappings` object[] — List of fields to rename. Not applicable for nested fields, but can be used to rename fields already flattened via dot notation. - `from_field` string, required — The field name in the source - `to_field` string, required — The field name to use in the destination - `omit_raw_text` boolean — Do not store the text that gets embedded along with the vector and the metadata in the destination. If set to true, only the vector and the metadata will be stored - in this case raw text for LLM use cases needs to be retrieved from another source. - `indexing` object, required — Indexing configuration - `host` string, required — The public endpoint of the Weaviate cluster. - `auth` union, required — Authentication method - object — Authenticate using an API token (suitable for Weaviate Cloud) - `mode` 'token', required - `token` string, required — API Token for the Weaviate instance - object — Authenticate using username and password (suitable for self-managed Weaviate clusters) - `mode` 'username_password', required - `username` string, required — Username for the Weaviate cluster - `password` string, required — Password for the Weaviate cluster - object — Do not authenticate (suitable for locally running test clusters, do not use for clusters with public IP addresses) - `mode` 'no_auth', required - `batch_size` integer — The number of records to send to Weaviate in each batch - `text_field` string — The field in the object that contains the embedded text - `tenant_id` string — The tenant ID to use for multi tenancy - `default_vectorizer` 'none' | 'text2vec-cohere' | 'text2vec-huggingface' | 'text2vec-openai' | 'text2vec-palm' | 'text2vec-contextionary' | 'text2vec-transformers' | 'text2vec-gpt4all' — The vectorizer to use if new classes need to be created - `additional_headers` object[] — Additional HTTP headers to send with every request. - `header_key` string, required - `value` string, required - `destinationType` 'weaviate', required - DestinationXata - `api_key` string, required — API Key to connect. - `db_url` string, required — URL pointing to your workspace. - `destinationType` 'xata', required ## Response `200` Successful operation - DestinationResponse — Provides details of a single destination. - `destinationId` string, UUID, required - `name` string, required - `destinationType` string, required - `workspaceId` string, UUID, required - `configuration` union, required — The values required to configure the destination. - DestinationGoogleSheets - `spreadsheet_id` string, required — The link to your spreadsheet. See this guide for more details. - `credentials` object, required — Google API Credentials for connecting to Google Sheets and Google Drive APIs - `client_id` string, required — The Client ID of your Google Sheets developer application. - `client_secret` string, required — The Client Secret of your Google Sheets developer application. - `refresh_token` string, required — The token for obtaining new access token. - `destinationType` 'google-sheets', required - DestinationAstra — The configuration model for the Vector DB based destinations. This model is used to generate the UI for the destination configuration, as well as to provide type safety for the configuration passed to the destination. The configuration model is composed of four parts: * Processing configuration * Embedding configuration * Indexing configuration * Advanced configuration Processing, embedding and advanced configuration are provided by this base class, while the indexing configuration is provided by the destination connector in the sub class. - `embedding` union, required — Embedding configuration - object — Use the OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `mode` 'openai', required - `openai_key` string, required - object — Use the Cohere API to embed text. - `mode` 'cohere', required - `cohere_key` string, required - object — Use a fake embedding made out of random vectors with 1536 embedding dimensions. This is useful for testing the data pipeline without incurring any costs. - `mode` 'fake', required - object — Use the Azure-hosted OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `mode` 'azure_openai', required - `openai_key` string, required — The API key for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `api_base` string, required — The base URL for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `deployment` string, required — The deployment for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - object — Use a service that's compatible with the OpenAI API to embed text. - `mode` 'openai_compatible', required - `api_key` string - `base_url` string, required — The base URL for your OpenAI-compatible service - `model_name` string — The name of the model to use for embedding - `dimensions` integer, required — The number of dimensions the embedding model is generating - `processing` object, required - `chunk_size` integer, required — Size of chunks in tokens to store in vector store (make sure it is not too big for the context if your LLM) - `chunk_overlap` integer — Size of overlap between chunks in tokens to store in vector store to better capture relevant context - `text_fields` string[] — List of fields in the record that should be used to calculate the embedding. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered text fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. - `metadata_fields` string[] — List of fields in the record that should be stored as metadata. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered metadata fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. When specifying nested paths, all matching values are flattened into an array set to a field named by the path. - `text_splitter` union — Split text fields into chunks based on the specified method. - object — Split the text by the list of separators until the chunk size is reached, using the earlier mentioned separators where possible. This is useful for splitting text fields by paragraphs, sentences, words, etc. - `mode` 'separator', required - `separators` string[] — List of separator strings to split text fields by. The separator itself needs to be wrapped in double quotes, e.g. to split by the dot character, use ".". To split by a newline, use "\n". - `keep_separator` boolean — Whether to keep the separator in the resulting chunks - object — Split the text by Markdown headers down to the specified header level. If the chunk size fits multiple sections, they will be combined into a single chunk. - `mode` 'markdown', required - `split_level` integer — Level of markdown headers to split text fields by. Headings down to the specified level will be used as split points - object — Split the text by suitable delimiters based on the programming language. This is useful for splitting code into chunks. - `mode` 'code', required - `language` 'cpp' | 'go' | 'java' | 'js' | 'php' | 'proto' | 'python' | 'rst' | 'ruby' | 'rust' | 'scala' | 'swift' | 'markdown' | 'latex' | 'html' | 'sol', required — Split code in suitable places based on the programming language - `field_name_mappings` object[] — List of fields to rename. Not applicable for nested fields, but can be used to rename fields already flattened via dot notation. - `from_field` string, required — The field name in the source - `to_field` string, required — The field name to use in the destination - `omit_raw_text` boolean — Do not store the text that gets embedded along with the vector and the metadata in the destination. If set to true, only the vector and the metadata will be stored - in this case raw text for LLM use cases needs to be retrieved from another source. - `indexing` object, required — Astra DB gives developers the APIs, real-time data and ecosystem integrations to put accurate RAG and Gen AI apps with fewer hallucinations in production. - `astra_db_app_token` string, required — The application token authorizes a user to connect to a specific Astra DB database. It is created when the user clicks the Generate Token button on the Overview tab of the Database page in the Astra UI. - `astra_db_endpoint` string, required — The endpoint specifies which Astra DB database queries are sent to. It can be copied from the Database Details section of the Overview tab of the Database page in the Astra UI. - `astra_db_keyspace` string, required — Keyspaces (or Namespaces) serve as containers for organizing data within a database. You can create a new keyspace uisng the Data Explorer tab in the Astra UI. The keyspace default_keyspace is created for you when you create a Vector Database in Astra DB. - `collection` string, required — Collections hold data. They are analagous to tables in traditional Cassandra terminology. This tool will create the collection with the provided name automatically if it does not already exist. Alternatively, you can create one thorugh the Data Explorer tab in the Astra UI. - `destinationType` 'astra', required - DestinationAwsDatalake - `aws_account_id` string — target aws account id - `credentials` union, required — Choose How to Authenticate to AWS. - object - `credentials_title` 'IAM Role', required — Name of the credentials - `role_arn` string, required — Will assume this role to write data to s3 - object - `credentials_title` 'IAM User', required — Name of the credentials - `aws_access_key_id` string, required — AWS User Access Key Id - `aws_secret_access_key` string, required — Secret Access Key - `region` '' | 'af-south-1' | 'ap-east-1' | 'ap-northeast-1' | 'ap-northeast-2' | 'ap-northeast-3' | 'ap-south-1' | 'ap-south-2' | 'ap-southeast-1' | 'ap-southeast-2' | 'ap-southeast-3' | 'ap-southeast-4' | 'ca-central-1' | 'ca-west-1' | 'cn-north-1' | 'cn-northwest-1' | 'eu-central-1' | 'eu-central-2' | 'eu-north-1' | 'eu-south-1' | 'eu-south-2' | 'eu-west-1' | 'eu-west-2' | 'eu-west-3' | 'il-central-1' | 'me-central-1' | 'me-south-1' | 'sa-east-1' | 'us-east-1' | 'us-east-2' | 'us-gov-east-1' | 'us-gov-west-1' | 'us-west-1' | 'us-west-2', required — The region of the S3 bucket. See here for all region codes. - `bucket_name` string, required — The name of the S3 bucket. Read more here. - `bucket_prefix` string — S3 prefix - `lakeformation_database_name` string, required — The default database this destination will use to create tables in per stream. Can be changed per connection by customizing the namespace. - `lakeformation_database_default_tag_key` string — Add a default tag key to databases created by this destination - `lakeformation_database_default_tag_values` string — Add default values for the `Tag Key` to databases created by this destination. Comma separate for multiple values. - `lakeformation_governed_tables` boolean — Whether to create tables as LF governed tables. - `format` union — Format of the data output. - object - `format_type` 'JSONL', required - `compression_codec` 'UNCOMPRESSED' | 'GZIP' — The compression algorithm used to compress data. - object - `format_type` 'Parquet', required - `compression_codec` 'UNCOMPRESSED' | 'SNAPPY' | 'GZIP' | 'ZSTD' — The compression algorithm used to compress data. - `partitioning` 'NO PARTITIONING' | 'DATE' | 'YEAR' | 'MONTH' | 'DAY' | 'YEAR/MONTH' | 'YEAR/MONTH/DAY' — Partition data by cursor fields when a cursor field is a date - `glue_catalog_float_as_decimal` boolean — Cast float/double as decimal(38,18). This can help achieve higher accuracy and represent numbers correctly as received from the source. - `destinationType` 'aws-datalake', required - DestinationAzureBlobStorage - `azure_blob_storage_endpoint_domain_name` string — This is Azure Blob Storage endpoint domain name. Leave default value (or leave it empty if run container from command line) to use Microsoft native from example. - `azure_blob_storage_container_name` string — The name of the Azure blob storage container. If not exists - will be created automatically. May be empty, then will be created automatically airbytecontainer+timestamp - `azure_blob_storage_account_name` string, required — The account's name of the Azure Blob Storage. - `azure_blob_storage_account_key` string, required — The Azure blob storage account key. - `azure_blob_storage_output_buffer_size` integer — The amount of megabytes to buffer for the output stream to Azure. This will impact memory footprint on workers, but may need adjustment for performance and appropriate block size in Azure. - `azure_blob_storage_spill_size` integer — The amount of megabytes after which the connector should spill the records in a new blob object. Make sure to configure size greater than individual records. Enter 0 if not applicable - `format` union, required — Output data format - object - `format_type` 'CSV', required - `flattening` 'No flattening' | 'Root level flattening', required — Whether the input json data should be normalized (flattened) in the output CSV. Please refer to docs for details. - object - `format_type` 'JSONL', required - `destinationType` 'azure-blob-storage', required - DestinationBigquery - `project_id` string, required — The GCP project ID for the project containing the target BigQuery dataset. Read more here. - `dataset_location` 'US' | 'EU' | 'asia-east1' | 'asia-east2' | 'asia-northeast1' | 'asia-northeast2' | 'asia-northeast3' | 'asia-south1' | 'asia-south2' | 'asia-southeast1' | 'asia-southeast2' | 'australia-southeast1' | 'australia-southeast2' | 'europe-central1' | 'europe-central2' | 'europe-north1' | 'europe-southwest1' | 'europe-west1' | 'europe-west2' | 'europe-west3' | 'europe-west4' | 'europe-west6' | 'europe-west7' | 'europe-west8' | 'europe-west9' | 'europe-west12' | 'me-central1' | 'me-central2' | 'me-west1' | 'northamerica-northeast1' | 'northamerica-northeast2' | 'southamerica-east1' | 'southamerica-west1' | 'us-central1' | 'us-east1' | 'us-east2' | 'us-east3' | 'us-east4' | 'us-east5' | 'us-south1' | 'us-west1' | 'us-west2' | 'us-west3' | 'us-west4', required — The location of the dataset. Warning: Changes made after creation will not be applied. Read more here. - `dataset_id` string, required — The default BigQuery Dataset ID that tables are replicated to if the source does not specify a namespace. Read more here. - `loading_method` union — The way data will be uploaded to BigQuery. - object — (recommended) Writes large batches of records to a file, uploads the file to GCS, then uses COPY INTO to load your data into BigQuery. Provides best-in-class speed, reliability and scalability. Read more about GCS Staging here. - `method` 'GCS Staging', required - `credential` object, required — An HMAC key is a type of credential and can be associated with a service account or a user account in Cloud Storage. Read more here. - `credential_type` 'HMAC_KEY', required - `hmac_key_access_id` string, required — HMAC key access ID. When linked to a service account, this ID is 61 characters long; when linked to a user account, it is 24 characters long. - `hmac_key_secret` string, required — The corresponding secret for the access ID. It is a 40-character base-64 encoded string. - `gcs_bucket_name` string, required — The name of the GCS bucket. Read more here. - `gcs_bucket_path` string, required — Directory under the GCS bucket where data will be written. - `keep_files_in_gcs-bucket` 'Delete all tmp files from GCS' | 'Keep all tmp files in GCS' — This upload method is supposed to temporary store records in GCS bucket. By this select you can chose if these records should be removed from GCS when migration has finished. The default "Delete all tmp files from GCS" value is used if not set explicitly. - object — (not recommended) Direct loading using SQL INSERT statements. This method is extremely inefficient and provided only for quick testing. In all other cases, you should use GCS staging. - `method` 'Standard', required - `credentials_json` string — The contents of the JSON service account key. Check out the docs if you need help generating this key. Default credentials will be used if this field is left empty. - `transformation_priority` 'interactive' | 'batch' — Interactive run type means that the query is executed as soon as possible, and these queries count towards concurrent rate limit and daily limit. Read more about interactive run type here. Batch queries are queued and started as soon as idle resources are available in the BigQuery shared resource pool, which usually occurs within a few minutes. Batch queries don’t count towards your concurrent rate limit. Read more about batch queries here. The default "interactive" value is used if not set explicitly. - `big_query_client_buffer_size_mb` integer — Google BigQuery client's chunk (buffer) size (MIN=1, MAX = 15) for each table. The size that will be written by a single RPC. Written data will be buffered and only flushed upon reaching this size or closing the channel. The default 15MB value is used if not set explicitly. Read more here. - `raw_data_dataset` string — The dataset to write raw tables into (default: airbyte_internal) - `disable_type_dedupe` boolean — Disable Writing Final Tables. WARNING! The data format in _airbyte_data is likely stable but there are no guarantees that other metadata columns will remain the same in future versions - `destinationType` 'bigquery', required - DestinationClickhouse - `host` string, required — Hostname of the database. - `port` integer, required — HTTP port of the database. - `database` string, required — Name of the database. - `username` string, required — Username to use to access the database. - `password` string — Password associated with the username. - `jdbc_url_params` string — Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (example: key1=value1&key2=value2&key3=value3). - `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use. - object - `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database - object - `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host. - `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa ) - object - `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host - `tunnel_user_password` string, required — OS-level password for logging into the jump server host - `destinationType` 'clickhouse', required - DestinationConvex - `deployment_url` string, required — URL of the Convex deployment that is the destination - `access_key` string, required — API access key used to send data to a Convex deployment. - `destinationType` 'convex', required - DestinationCumulio - `api_host` string, required — URL of the Cumul.io API (e.g. 'https://api.cumul.io', 'https://api.us.cumul.io', or VPC-specific API url). Defaults to 'https://api.cumul.io'. - `api_key` string, required — An API key generated in Cumul.io's platform (can be generated here: https://app.cumul.io/start/profile/integration). - `api_token` string, required — The corresponding API token generated in Cumul.io's platform (can be generated here: https://app.cumul.io/start/profile/integration). - `destinationType` 'cumulio', required - DestinationDatabend - `host` string, required — Hostname of the database. - `port` integer — Port of the database. - `database` string, required — Name of the database. - `table` string — The default table was written to. - `username` string, required — Username to use to access the database. - `password` string — Password associated with the username. - `destinationType` 'databend', required - DestinationDatabricks - `accept_terms` boolean, required — You must agree to the Databricks JDBC Driver Terms & Conditions to use this connector. - `databricks_server_hostname` string, required — Databricks Cluster Server Hostname. - `databricks_http_path` string, required — Databricks Cluster HTTP Path. - `databricks_port` string — Databricks Cluster Port. - `databricks_personal_access_token` string, required — Databricks Personal Access Token for making authenticated requests. - `database` string — The name of the catalog. If not specified otherwise, the "hive_metastore" will be used. - `schema` string — The default schema tables are written. If not specified otherwise, the "default" will be used. - `enable_schema_evolution` boolean — Support schema evolution for all streams. If "false", the connector might fail when a stream's schema changes. - `data_source` union, required — Storage on which the delta lake is built. - object - `data_source_type` 'MANAGED_TABLES_STORAGE', required - object - `data_source_type` 'S3_STORAGE', required - `s3_bucket_name` string, required — The name of the S3 bucket to use for intermittent staging of the data. - `s3_bucket_path` string, required — The directory under the S3 bucket where data will be written. - `s3_bucket_region` '' | 'us-east-1' | 'us-east-2' | 'us-west-1' | 'us-west-2' | 'af-south-1' | 'ap-east-1' | 'ap-south-1' | 'ap-northeast-1' | 'ap-northeast-2' | 'ap-northeast-3' | 'ap-southeast-1' | 'ap-southeast-2' | 'ca-central-1' | 'cn-north-1' | 'cn-northwest-1' | 'eu-central-1' | 'eu-north-1' | 'eu-south-1' | 'eu-west-1' | 'eu-west-2' | 'eu-west-3' | 'sa-east-1' | 'me-south-1' | 'us-gov-east-1' | 'us-gov-west-1', required — The region of the S3 staging bucket to use if utilising a copy strategy. - `s3_access_key_id` string, required — The Access Key Id granting allow one to access the above S3 staging bucket. Airbyte requires Read and Write permissions to the given bucket. - `s3_secret_access_key` string, required — The corresponding secret to the above access key id. - `file_name_pattern` string — The pattern allows you to set the file-name format for the S3 staging file(s) - object - `data_source_type` 'AZURE_BLOB_STORAGE', required - `azure_blob_storage_endpoint_domain_name` string — This is Azure Blob Storage endpoint domain name. Leave default value (or leave it empty if run container from command line) to use Microsoft native from example. - `azure_blob_storage_account_name` string, required — The account's name of the Azure Blob Storage. - `azure_blob_storage_container_name` string, required — The name of the Azure blob storage container. - `azure_blob_storage_sas_token` string, required — Shared access signature (SAS) token to grant limited access to objects in your storage account. - `purge_staging_data` boolean — Default to 'true'. Switch it to 'false' for debugging purpose. - `destinationType` 'databricks', required - DestinationDevNull - `test_destination` object, required — The type of destination to be used - `test_destination_type` 'SILENT', required - `destinationType` 'dev-null', required - DestinationDuckdb - `motherduck_api_key` string — API key to use for authentication to a MotherDuck database. - `destination_path` string, required — Path to the .duckdb file, or the text 'md:' to connect to MotherDuck. The file will be placed inside that local mount. For more information check out our docs - `schema` string — Database schema name, default for duckdb is 'main'. - `destinationType` 'duckdb', required - DestinationDynamodb - `dynamodb_endpoint` string — This is your DynamoDB endpoint url.(if you are working with AWS DynamoDB, just leave empty). - `dynamodb_table_name_prefix` string, required — The prefix to use when naming DynamoDB tables. - `dynamodb_region` '' | 'af-south-1' | 'ap-east-1' | 'ap-northeast-1' | 'ap-northeast-2' | 'ap-northeast-3' | 'ap-south-1' | 'ap-south-2' | 'ap-southeast-1' | 'ap-southeast-2' | 'ap-southeast-3' | 'ap-southeast-4' | 'ca-central-1' | 'ca-west-1' | 'cn-north-1' | 'cn-northwest-1' | 'eu-central-1' | 'eu-central-2' | 'eu-north-1' | 'eu-south-1' | 'eu-south-2' | 'eu-west-1' | 'eu-west-2' | 'eu-west-3' | 'il-central-1' | 'me-central-1' | 'me-south-1' | 'sa-east-1' | 'us-east-1' | 'us-east-2' | 'us-gov-east-1' | 'us-gov-west-1' | 'us-west-1' | 'us-west-2', required — The region of the DynamoDB. - `access_key_id` string, required — The access key id to access the DynamoDB. Airbyte requires Read and Write permissions to the DynamoDB. - `secret_access_key` string, required — The corresponding secret to the access key id. - `destinationType` 'dynamodb', required - DestinationElasticsearch - `endpoint` string, required — The full url of the Elasticsearch server - `upsert` boolean — If a primary key identifier is defined in the source, an upsert will be performed using the primary key value as the elasticsearch doc id. Does not support composite primary keys. - `ca_certificate` string — CA certificate - `authenticationMethod` union — The type of authentication to be used - object — Use a api key and secret combination to authenticate - `method` 'secret', required - `apiKeyId` string, required — The Key ID to used when accessing an enterprise Elasticsearch instance. - `apiKeySecret` string, required — The secret associated with the API Key ID. - object — Basic auth header with a username and password - `method` 'basic', required - `username` string, required — Basic auth username to access a secure Elasticsearch server - `password` string, required — Basic auth password to access a secure Elasticsearch server - `destinationType` 'elasticsearch', required - DestinationFirebolt - `username` string, required — Firebolt email address you use to login. - `password` string, required — Firebolt password. - `account` string — Firebolt account to login. - `host` string — The host name of your Firebolt database. - `database` string, required — The database to connect to. - `engine` string — Engine name or url to connect to. - `loading_method` union — Loading method used to select the way data will be uploaded to Firebolt - object - `method` 'SQL', required - object - `method` 'S3', required - `s3_bucket` string, required — The name of the S3 bucket. - `s3_region` string, required — Region name of the S3 bucket. - `aws_key_id` string, required — AWS access key granting read and write access to S3. - `aws_key_secret` string, required — Corresponding secret part of the AWS Key - `destinationType` 'firebolt', required - DestinationFirestore - `project_id` string, required — The GCP project ID for the project containing the target BigQuery dataset. - `credentials_json` string — The contents of the JSON service account key. Check out the docs if you need help generating this key. Default credentials will be used if this field is left empty. - `destinationType` 'firestore', required - DestinationGcs - `gcs_bucket_name` string, required — You can find the bucket name in the App Engine Admin console Application Settings page, under the label Google Cloud Storage Bucket. Read more here. - `gcs_bucket_path` string, required — GCS Bucket Path string Subdirectory under the above bucket to sync the data into. - `gcs_bucket_region` 'northamerica-northeast1' | 'northamerica-northeast2' | 'us-central1' | 'us-east1' | 'us-east4' | 'us-west1' | 'us-west2' | 'us-west3' | 'us-west4' | 'southamerica-east1' | 'southamerica-west1' | 'europe-central2' | 'europe-north1' | 'europe-west1' | 'europe-west2' | 'europe-west3' | 'europe-west4' | 'europe-west6' | 'asia-east1' | 'asia-east2' | 'asia-northeast1' | 'asia-northeast2' | 'asia-northeast3' | 'asia-south1' | 'asia-south2' | 'asia-southeast1' | 'asia-southeast2' | 'australia-southeast1' | 'australia-southeast2' | 'asia' | 'eu' | 'us' | 'asia1' | 'eur4' | 'nam4' — Select a Region of the GCS Bucket. Read more here. - `credential` object, required — An HMAC key is a type of credential and can be associated with a service account or a user account in Cloud Storage. Read more here. - `credential_type` 'HMAC_KEY', required - `hmac_key_access_id` string, required — When linked to a service account, this ID is 61 characters long; when linked to a user account, it is 24 characters long. Read more here. - `hmac_key_secret` string, required — The corresponding secret for the access ID. It is a 40-character base-64 encoded string. Read more here. - `format` union, required — Output data format. One of the following formats must be selected - AVRO format, PARQUET format, CSV format, or JSONL format. - object - `format_type` 'Avro', required - `compression_codec` union, required — The compression algorithm used to compress data. Default to no compression. - object - … - object - … - object - … - object - … - object - … - object - … - object - `format_type` 'CSV', required - `flattening` 'No flattening' | 'Root level flattening' — Whether the input JSON data should be normalized (flattened) in the output CSV. Please refer to docs for details. - `compression` union — Whether the output files should be compressed. If compression is selected, the output filename will have an extra extension (GZIP: ".csv.gz"). - object - … - object - … - object - `format_type` 'JSONL', required - `compression` union — Whether the output files should be compressed. If compression is selected, the output filename will have an extra extension (GZIP: ".jsonl.gz"). - object - … - object - … - object - `format_type` 'Parquet', required - `compression_codec` 'UNCOMPRESSED' | 'SNAPPY' | 'GZIP' | 'LZO' | 'BROTLI' | 'LZ4' | 'ZSTD' — The compression algorithm used to compress data pages. - `block_size_mb` integer — This is the size of a row group being buffered in memory. It limits the memory usage when writing. Larger values will improve the IO when reading, but consume more memory when writing. Default: 128 MB. - `max_padding_size_mb` integer — Maximum size allowed as padding to align row groups. This is also the minimum size of a row group. Default: 8 MB. - `page_size_kb` integer — The page size is for compression. A block is composed of pages. A page is the smallest unit that must be read fully to access a single record. If this value is too small, the compression will deteriorate. Default: 1024 KB. - `dictionary_page_size_kb` integer — There is one dictionary page per column per row group when dictionary encoding is used. The dictionary page size works like the page size but for dictionary. Default: 1024 KB. - `dictionary_encoding` boolean — Default: true. - `destinationType` 'gcs', required - DestinationKeen - `project_id` string, required — To get Keen Project ID, navigate to the Access tab from the left-hand, side panel and check the Project Details section. - `api_key` string, required — To get Keen Master API Key, navigate to the Access tab from the left-hand, side panel and check the Project Details section. - `infer_timestamp` boolean — Allow connector to guess keen.timestamp value based on the streamed data. - `destinationType` 'keen', required - DestinationKinesis - `endpoint` string, required — AWS Kinesis endpoint. - `region` string, required — AWS region. Your account determines the Regions that are available to you. - `shardCount` integer, required — Number of shards to which the data should be streamed. - `accessKey` string, required — Generate the AWS Access Key for current user. - `privateKey` string, required — The AWS Private Key - a string of numbers and letters that are unique for each account, also known as a "recovery phrase". - `bufferSize` integer, required — Buffer size for storing kinesis records before being batch streamed. - `destinationType` 'kinesis', required - DestinationLangchain - `processing` object, required - `chunk_size` integer, required — Size of chunks in tokens to store in vector store (make sure it is not too big for the context if your LLM) - `chunk_overlap` integer — Size of overlap between chunks in tokens to store in vector store to better capture relevant context - `text_fields` string[], required — List of fields in the record that should be used to calculate the embedding. All other fields are passed along as meta fields. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered text fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. - `embedding` union, required — Embedding configuration - object — Use the OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `mode` 'openai' - `openai_key` string, required - object — Use a fake embedding made out of random vectors with 1536 embedding dimensions. This is useful for testing the data pipeline without incurring any costs. - `mode` 'fake' - `indexing` union, required — Indexing configuration - object — Pinecone is a popular vector store that can be used to store and retrieve embeddings. It is a managed service and can also be queried from outside of langchain. - `mode` 'pinecone' - `pinecone_key` string, required - `pinecone_environment` string, required — Pinecone environment to use - `index` string, required — Pinecone index to use - object — DocArrayHnswSearch is a lightweight Document Index implementation provided by Docarray that runs fully locally and is best suited for small- to medium-sized datasets. It stores vectors on disk in hnswlib, and stores all other data in SQLite. - `mode` 'DocArrayHnswSearch' - `destination_path` string, required — Path to the directory where hnswlib and meta data files will be written. The files will be placed inside that local mount. All files in the specified destination directory will be deleted on each run. - object — Chroma is a popular vector store that can be used to store and retrieve embeddings. It will build its index in memory and persist it to disk by the end of the sync. - `mode` 'chroma_local' - `destination_path` string, required — Path to the directory where chroma files will be written. The files will be placed inside that local mount. - `collection_name` string — Name of the collection to use. - `destinationType` 'langchain', required - DestinationMilvus — The configuration model for the Vector DB based destinations. This model is used to generate the UI for the destination configuration, as well as to provide type safety for the configuration passed to the destination. The configuration model is composed of four parts: * Processing configuration * Embedding configuration * Indexing configuration * Advanced configuration Processing, embedding and advanced configuration are provided by this base class, while the indexing configuration is provided by the destination connector in the sub class. - `embedding` union, required — Embedding configuration - object — Use the OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `mode` 'openai', required - `openai_key` string, required - object — Use the Cohere API to embed text. - `mode` 'cohere', required - `cohere_key` string, required - object — Use a fake embedding made out of random vectors with 1536 embedding dimensions. This is useful for testing the data pipeline without incurring any costs. - `mode` 'fake', required - object — Use the Azure-hosted OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `mode` 'azure_openai', required - `openai_key` string, required — The API key for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `api_base` string, required — The base URL for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `deployment` string, required — The deployment for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - object — Use a service that's compatible with the OpenAI API to embed text. - `mode` 'openai_compatible', required - `api_key` string - `base_url` string, required — The base URL for your OpenAI-compatible service - `model_name` string — The name of the model to use for embedding - `dimensions` integer, required — The number of dimensions the embedding model is generating - `processing` object, required - `chunk_size` integer, required — Size of chunks in tokens to store in vector store (make sure it is not too big for the context if your LLM) - `chunk_overlap` integer — Size of overlap between chunks in tokens to store in vector store to better capture relevant context - `text_fields` string[] — List of fields in the record that should be used to calculate the embedding. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered text fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. - `metadata_fields` string[] — List of fields in the record that should be stored as metadata. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered metadata fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. When specifying nested paths, all matching values are flattened into an array set to a field named by the path. - `text_splitter` union — Split text fields into chunks based on the specified method. - object — Split the text by the list of separators until the chunk size is reached, using the earlier mentioned separators where possible. This is useful for splitting text fields by paragraphs, sentences, words, etc. - `mode` 'separator', required - `separators` string[] — List of separator strings to split text fields by. The separator itself needs to be wrapped in double quotes, e.g. to split by the dot character, use ".". To split by a newline, use "\n". - `keep_separator` boolean — Whether to keep the separator in the resulting chunks - object — Split the text by Markdown headers down to the specified header level. If the chunk size fits multiple sections, they will be combined into a single chunk. - `mode` 'markdown', required - `split_level` integer — Level of markdown headers to split text fields by. Headings down to the specified level will be used as split points - object — Split the text by suitable delimiters based on the programming language. This is useful for splitting code into chunks. - `mode` 'code', required - `language` 'cpp' | 'go' | 'java' | 'js' | 'php' | 'proto' | 'python' | 'rst' | 'ruby' | 'rust' | 'scala' | 'swift' | 'markdown' | 'latex' | 'html' | 'sol', required — Split code in suitable places based on the programming language - `field_name_mappings` object[] — List of fields to rename. Not applicable for nested fields, but can be used to rename fields already flattened via dot notation. - `from_field` string, required — The field name in the source - `to_field` string, required — The field name to use in the destination - `omit_raw_text` boolean — Do not store the text that gets embedded along with the vector and the metadata in the destination. If set to true, only the vector and the metadata will be stored - in this case raw text for LLM use cases needs to be retrieved from another source. - `indexing` object, required — Indexing configuration - `host` string, required — The public endpoint of the Milvus instance. - `db` string — The database to connect to - `collection` string, required — The collection to load data into - `auth` union, required — Authentication method - object — Authenticate using an API token (suitable for Zilliz Cloud) - `mode` 'token', required - `token` string, required — API Token for the Milvus instance - object — Authenticate using username and password (suitable for self-managed Milvus clusters) - `mode` 'username_password', required - `username` string, required — Username for the Milvus instance - `password` string, required — Password for the Milvus instance - object — Do not authenticate (suitable for locally running test clusters, do not use for clusters with public IP addresses) - `mode` 'no_auth', required - `vector_field` string — The field in the entity that contains the vector - `text_field` string — The field in the entity that contains the embedded text - `destinationType` 'milvus', required - DestinationMongodb - `instance_type` union — MongoDb instance to connect to. For MongoDB Atlas and Replica Set TLS connection is used by default. - object - `instance` 'standalone', required - `host` string, required — The Host of a Mongo database to be replicated. - `port` integer, required — The Port of a Mongo database to be replicated. - object - `instance` 'replica', required - `server_addresses` string, required — The members of a replica set. Please specify `host`:`port` of each member seperated by comma. - `replica_set` string — A replica set name. - object - `instance` 'atlas', required - `cluster_url` string, required — URL of a cluster to connect to. - `database` string, required — Name of the database. - `auth_type` union, required — Authorization type. - object — None. - `authorization` 'none', required - object — Login/Password. - `authorization` 'login/password', required - `username` string, required — Username to use to access the database. - `password` string, required — Password associated with the username. - `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use. - object - `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database - object - `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host. - `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa ) - object - `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host - `tunnel_user_password` string, required — OS-level password for logging into the jump server host - `destinationType` 'mongodb', required - DestinationMssql - `host` string, required — The host name of the MSSQL database. - `port` integer, required — The port of the MSSQL database. - `database` string, required — The name of the MSSQL database. - `schema` string, required — The default schema tables are written to if the source does not specify a namespace. The usual value for this field is "public". - `username` string, required — The username which is used to access the database. - `password` string — The password associated with this username. - `jdbc_url_params` string — Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (example: key1=value1&key2=value2&key3=value3). - `ssl_method` union — The encryption method which is used to communicate with the database. - object — Use the certificate provided by the server without verification. (For testing purposes only!) - `ssl_method` 'encrypted_trust_server_certificate', required - object — Verify and use the certificate provided by the server. - `ssl_method` 'encrypted_verify_certificate', required - `hostNameInCertificate` string — Specifies the host name of the server. The value of this property must match the subject property of the certificate. - `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use. - object - `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database - object - `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host. - `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa ) - object - `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host - `tunnel_user_password` string, required — OS-level password for logging into the jump server host - `destinationType` 'mssql', required - DestinationMysql - `host` string, required — Hostname of the database. - `port` integer, required — Port of the database. - `database` string, required — Name of the database. - `username` string, required — Username to use to access the database. - `password` string — Password associated with the username. - `jdbc_url_params` string — Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (example: key1=value1&key2=value2&key3=value3). - `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use. - object - `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database - object - `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host. - `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa ) - object - `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host - `tunnel_user_password` string, required — OS-level password for logging into the jump server host - `destinationType` 'mysql', required - DestinationOracle - `host` string, required — The hostname of the database. - `port` integer, required — The port of the database. - `sid` string, required — The System Identifier uniquely distinguishes the instance from any other instance on the same computer. - `username` string, required — The username to access the database. This user must have CREATE USER privileges in the database. - `password` string — The password associated with the username. - `jdbc_url_params` string — Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (example: key1=value1&key2=value2&key3=value3). - `schema` string — The default schema is used as the target schema for all statements issued from the connection that do not explicitly specify a schema name. The usual value for this field is "airbyte". In Oracle, schemas and users are the same thing, so the "user" parameter is used as the login credentials and this is used for the default Airbyte message schema. - `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use. - object - `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database - object - `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host. - `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa ) - object - `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host - `tunnel_user_password` string, required — OS-level password for logging into the jump server host - `destinationType` 'oracle', required - DestinationPinecone — The configuration model for the Vector DB based destinations. This model is used to generate the UI for the destination configuration, as well as to provide type safety for the configuration passed to the destination. The configuration model is composed of four parts: * Processing configuration * Embedding configuration * Indexing configuration * Advanced configuration Processing, embedding and advanced configuration are provided by this base class, while the indexing configuration is provided by the destination connector in the sub class. - `embedding` union, required — Embedding configuration - object — Use the OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `mode` 'openai', required - `openai_key` string, required - object — Use the Cohere API to embed text. - `mode` 'cohere', required - `cohere_key` string, required - object — Use a fake embedding made out of random vectors with 1536 embedding dimensions. This is useful for testing the data pipeline without incurring any costs. - `mode` 'fake', required - object — Use the Azure-hosted OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `mode` 'azure_openai', required - `openai_key` string, required — The API key for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `api_base` string, required — The base URL for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `deployment` string, required — The deployment for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - object — Use a service that's compatible with the OpenAI API to embed text. - `mode` 'openai_compatible', required - `api_key` string - `base_url` string, required — The base URL for your OpenAI-compatible service - `model_name` string — The name of the model to use for embedding - `dimensions` integer, required — The number of dimensions the embedding model is generating - `processing` object, required - `chunk_size` integer, required — Size of chunks in tokens to store in vector store (make sure it is not too big for the context if your LLM) - `chunk_overlap` integer — Size of overlap between chunks in tokens to store in vector store to better capture relevant context - `text_fields` string[] — List of fields in the record that should be used to calculate the embedding. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered text fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. - `metadata_fields` string[] — List of fields in the record that should be stored as metadata. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered metadata fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. When specifying nested paths, all matching values are flattened into an array set to a field named by the path. - `text_splitter` union — Split text fields into chunks based on the specified method. - object — Split the text by the list of separators until the chunk size is reached, using the earlier mentioned separators where possible. This is useful for splitting text fields by paragraphs, sentences, words, etc. - `mode` 'separator', required - `separators` string[] — List of separator strings to split text fields by. The separator itself needs to be wrapped in double quotes, e.g. to split by the dot character, use ".". To split by a newline, use "\n". - `keep_separator` boolean — Whether to keep the separator in the resulting chunks - object — Split the text by Markdown headers down to the specified header level. If the chunk size fits multiple sections, they will be combined into a single chunk. - `mode` 'markdown', required - `split_level` integer — Level of markdown headers to split text fields by. Headings down to the specified level will be used as split points - object — Split the text by suitable delimiters based on the programming language. This is useful for splitting code into chunks. - `mode` 'code', required - `language` 'cpp' | 'go' | 'java' | 'js' | 'php' | 'proto' | 'python' | 'rst' | 'ruby' | 'rust' | 'scala' | 'swift' | 'markdown' | 'latex' | 'html' | 'sol', required — Split code in suitable places based on the programming language - `field_name_mappings` object[] — List of fields to rename. Not applicable for nested fields, but can be used to rename fields already flattened via dot notation. - `from_field` string, required — The field name in the source - `to_field` string, required — The field name to use in the destination - `omit_raw_text` boolean — Do not store the text that gets embedded along with the vector and the metadata in the destination. If set to true, only the vector and the metadata will be stored - in this case raw text for LLM use cases needs to be retrieved from another source. - `indexing` object, required — Pinecone is a popular vector store that can be used to store and retrieve embeddings. - `pinecone_key` string, required — The Pinecone API key to use matching the environment (copy from Pinecone console) - `pinecone_environment` string, required — Pinecone Cloud environment to use - `index` string, required — Pinecone index in your project to load data into - `destinationType` 'pinecone', required - DestinationPostgres - `host` string, required — Hostname of the database. - `port` integer, required — Port of the database. - `database` string, required — Name of the database. - `schema` string, required — The default schema tables are written to if the source does not specify a namespace. The usual value for this field is "public". - `username` string, required — Username to use to access the database. - `password` string — Password associated with the username. - `ssl_mode` union — SSL connection modes. disable - Chose this mode to disable encryption of communication between Airbyte and destination database allow - Chose this mode to enable encryption only when required by the source database prefer - Chose this mode to allow unencrypted connection only if the source database does not support encryption require - Chose this mode to always require encryption. If the source database server does not support encryption, connection will fail verify-ca - Chose this mode to always require encryption and to verify that the source database server has a valid SSL certificate verify-full - This is the most secure mode. Chose this mode to always require encryption and to verify the identity of the source database server See more information - in the docs. - object — Disable SSL. - `mode` 'disable', required - object — Allow SSL mode. - `mode` 'allow', required - object — Prefer SSL mode. - `mode` 'prefer', required - object — Require SSL mode. - `mode` 'require', required - object — Verify-ca SSL mode. - `mode` 'verify-ca', required - `ca_certificate` string, required — CA certificate - `client_key_password` string — Password for keystorage. This field is optional. If you do not add it - the password will be generated automatically. - object — Verify-full SSL mode. - `mode` 'verify-full', required - `ca_certificate` string, required — CA certificate - `client_certificate` string, required — Client certificate - `client_key` string, required — Client key - `client_key_password` string — Password for keystorage. This field is optional. If you do not add it - the password will be generated automatically. - `jdbc_url_params` string — Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (example: key1=value1&key2=value2&key3=value3). - `raw_data_schema` string — The schema to write raw tables into - `disable_type_dedupe` boolean — Disable Writing Final Tables. WARNING! The data format in _airbyte_data is likely stable but there are no guarantees that other metadata columns will remain the same in future versions - `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use. - object - `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database - object - `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host. - `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa ) - object - `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host - `tunnel_user_password` string, required — OS-level password for logging into the jump server host - `destinationType` 'postgres', required - DestinationPubsub - `project_id` string, required — The GCP project ID for the project containing the target PubSub. - `topic_id` string, required — The PubSub topic ID in the given GCP project ID. - `credentials_json` string, required — The contents of the JSON service account key. Check out the docs if you need help generating this key. - `ordering_enabled` boolean, required — If TRUE PubSub publisher will have message ordering enabled. Every message will have an ordering key of stream - `batching_enabled` boolean, required — If TRUE messages will be buffered instead of sending them one by one - `batching_delay_threshold` integer — Number of ms before the buffer is flushed - `batching_element_count_threshold` integer — Number of messages before the buffer is flushed - `batching_request_bytes_threshold` integer — Number of bytes before the buffer is flushed - `destinationType` 'pubsub', required - DestinationQdrant — The configuration model for the Vector DB based destinations. This model is used to generate the UI for the destination configuration, as well as to provide type safety for the configuration passed to the destination. The configuration model is composed of four parts: * Processing configuration * Embedding configuration * Indexing configuration * Advanced configuration Processing, embedding and advanced configuration are provided by this base class, while the indexing configuration is provided by the destination connector in the sub class. - `embedding` union, required — Embedding configuration - object — Use the OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `mode` 'openai', required - `openai_key` string, required - object — Use the Cohere API to embed text. - `mode` 'cohere', required - `cohere_key` string, required - object — Use a fake embedding made out of random vectors with 1536 embedding dimensions. This is useful for testing the data pipeline without incurring any costs. - `mode` 'fake', required - object — Use the Azure-hosted OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `mode` 'azure_openai', required - `openai_key` string, required — The API key for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `api_base` string, required — The base URL for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `deployment` string, required — The deployment for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - object — Use a service that's compatible with the OpenAI API to embed text. - `mode` 'openai_compatible', required - `api_key` string - `base_url` string, required — The base URL for your OpenAI-compatible service - `model_name` string — The name of the model to use for embedding - `dimensions` integer, required — The number of dimensions the embedding model is generating - `processing` object, required - `chunk_size` integer, required — Size of chunks in tokens to store in vector store (make sure it is not too big for the context if your LLM) - `chunk_overlap` integer — Size of overlap between chunks in tokens to store in vector store to better capture relevant context - `text_fields` string[] — List of fields in the record that should be used to calculate the embedding. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered text fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. - `metadata_fields` string[] — List of fields in the record that should be stored as metadata. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered metadata fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. When specifying nested paths, all matching values are flattened into an array set to a field named by the path. - `text_splitter` union — Split text fields into chunks based on the specified method. - object — Split the text by the list of separators until the chunk size is reached, using the earlier mentioned separators where possible. This is useful for splitting text fields by paragraphs, sentences, words, etc. - `mode` 'separator', required - `separators` string[] — List of separator strings to split text fields by. The separator itself needs to be wrapped in double quotes, e.g. to split by the dot character, use ".". To split by a newline, use "\n". - `keep_separator` boolean — Whether to keep the separator in the resulting chunks - object — Split the text by Markdown headers down to the specified header level. If the chunk size fits multiple sections, they will be combined into a single chunk. - `mode` 'markdown', required - `split_level` integer — Level of markdown headers to split text fields by. Headings down to the specified level will be used as split points - object — Split the text by suitable delimiters based on the programming language. This is useful for splitting code into chunks. - `mode` 'code', required - `language` 'cpp' | 'go' | 'java' | 'js' | 'php' | 'proto' | 'python' | 'rst' | 'ruby' | 'rust' | 'scala' | 'swift' | 'markdown' | 'latex' | 'html' | 'sol', required — Split code in suitable places based on the programming language - `field_name_mappings` object[] — List of fields to rename. Not applicable for nested fields, but can be used to rename fields already flattened via dot notation. - `from_field` string, required — The field name in the source - `to_field` string, required — The field name to use in the destination - `omit_raw_text` boolean — Do not store the text that gets embedded along with the vector and the metadata in the destination. If set to true, only the vector and the metadata will be stored - in this case raw text for LLM use cases needs to be retrieved from another source. - `indexing` object, required — Indexing configuration - `url` string, required — Public Endpoint of the Qdrant cluser - `auth_method` union — Method to authenticate with the Qdrant Instance - object - `mode` 'api_key_auth' - `api_key` string, required — API Key for the Qdrant instance - object - `mode` 'no_auth' - `prefer_grpc` boolean — Whether to prefer gRPC over HTTP. Set to true for Qdrant cloud clusters - `collection` string, required — The collection to load data into - `distance_metric` 'dot' | 'cos' | 'euc' — The Distance metric used to measure similarities among vectors. This field is only used if the collection defined in the does not exist yet and is created automatically by the connector. - `text_field` string — The field in the payload that contains the embedded text - `destinationType` 'qdrant', required - DestinationRedis - `host` string, required — Redis host to connect to. - `port` integer, required — Port of Redis. - `username` string, required — Username associated with Redis. - `password` string — Password associated with Redis. - `ssl` boolean — Indicates whether SSL encryption protocol will be used to connect to Redis. It is recommended to use SSL connection if possible. - `ssl_mode` union — SSL connection modes.
  • verify-full - This is the most secure mode. Always require encryption and verifies the identity of the source database server - object — Disable SSL. - `mode` 'disable', required - object — Verify-full SSL mode. - `mode` 'verify-full', required - `ca_certificate` string, required — CA certificate - `client_certificate` string, required — Client certificate - `client_key` string, required — Client key - `client_key_password` string — Password for keystorage. If you do not add it - the password will be generated automatically. - `cache_type` 'hash', required — Redis cache type to store data in. - `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use. - object - `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database - object - `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host. - `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa ) - object - `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host - `tunnel_user_password` string, required — OS-level password for logging into the jump server host - `destinationType` 'redis', required - DestinationRedshift - `host` string, required — Host Endpoint of the Redshift Cluster (must include the cluster-id, region and end with .redshift.amazonaws.com) - `port` integer, required — Port of the database. - `username` string, required — Username to use to access the database. - `password` string, required — Password associated with the username. - `database` string, required — Name of the database. - `schema` string, required — The default schema tables are written to if the source does not specify a namespace. Unless specifically configured, the usual value for this field is "public". - `jdbc_url_params` string — Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (example: key1=value1&key2=value2&key3=value3). - `uploading_method` union — The way data will be uploaded to Redshift. - object — (recommended) Uploads data to S3 and then uses a COPY to insert the data into Redshift. COPY is recommended for production workloads for better speed and scalability. See AWS docs for more details. - `method` 'S3 Staging', required - `s3_bucket_name` string, required — The name of the staging S3 bucket. - `s3_bucket_path` string — The directory under the S3 bucket where data will be written. If not provided, then defaults to the root directory. See path's name recommendations for more details. - `s3_bucket_region` '' | 'af-south-1' | 'ap-east-1' | 'ap-northeast-1' | 'ap-northeast-2' | 'ap-northeast-3' | 'ap-south-1' | 'ap-south-2' | 'ap-southeast-1' | 'ap-southeast-2' | 'ap-southeast-3' | 'ap-southeast-4' | 'ca-central-1' | 'ca-west-1' | 'cn-north-1' | 'cn-northwest-1' | 'eu-central-1' | 'eu-central-2' | 'eu-north-1' | 'eu-south-1' | 'eu-south-2' | 'eu-west-1' | 'eu-west-2' | 'eu-west-3' | 'il-central-1' | 'me-central-1' | 'me-south-1' | 'sa-east-1' | 'sa-east-1' | 'us-east-1' | 'us-east-2' | 'us-gov-east-1' | 'us-gov-west-1' | 'us-west-1' | 'us-west-2', required — The region of the S3 staging bucket. - `access_key_id` string, required — This ID grants access to the above S3 staging bucket. Airbyte requires Read and Write permissions to the given bucket. See AWS docs on how to generate an access key ID and secret access key. - `secret_access_key` string, required — The corresponding secret to the above access key id. See AWS docs on how to generate an access key ID and secret access key. - `file_name_pattern` string — The pattern allows you to set the file-name format for the S3 staging file(s) - `purge_staging_data` boolean — Whether to delete the staging files from S3 after completing the sync. See docs for details. - `encryption` union — How to encrypt the staging data - object — Staging data will be stored in plaintext. - … - object — Staging data will be encrypted using AES-CBC envelope encryption. - … - `file_buffer_count` integer — Number of file buffers allocated for writing data. Increasing this number is beneficial for connections using Change Data Capture (CDC) and up to the number of streams within a connection. Increasing the number of file buffers past the maximum number of streams has deteriorating effects - object — (not recommended) Direct loading using SQL INSERT statements. This method is extremely inefficient and provided only for quick testing. In all other cases, you should use S3 uploading. - `method` 'Standard', required - `raw_data_schema` string — The schema to write raw tables into - `enable_incremental_final_table_updates` boolean — When enabled your data will load into your final tables incrementally while your data is still being synced. When Disabled (the default), your data loads into your final tables once at the end of a sync. Note that this option only applies if you elect to create Final tables - `disable_type_dedupe` boolean — Disable Writing Final Tables. WARNING! The data format in _airbyte_data is likely stable but there are no guarantees that other metadata columns will remain the same in future versions - `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use. - object - `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database - object - `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host. - `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa ) - object - `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host - `tunnel_user_password` string, required — OS-level password for logging into the jump server host - `destinationType` 'redshift', required - DestinationS3 - `access_key_id` string — The access key ID to access the S3 bucket. Airbyte requires Read and Write permissions to the given bucket. Read more here. - `secret_access_key` string — The corresponding secret to the access key ID. Read more here - `s3_bucket_name` string, required — The name of the S3 bucket. Read more here. - `s3_bucket_path` string, required — Directory under the S3 bucket where data will be written. Read more here - `s3_bucket_region` '' | 'af-south-1' | 'ap-east-1' | 'ap-northeast-1' | 'ap-northeast-2' | 'ap-northeast-3' | 'ap-south-1' | 'ap-south-2' | 'ap-southeast-1' | 'ap-southeast-2' | 'ap-southeast-3' | 'ap-southeast-4' | 'ca-central-1' | 'ca-west-1' | 'cn-north-1' | 'cn-northwest-1' | 'eu-central-1' | 'eu-central-2' | 'eu-north-1' | 'eu-south-1' | 'eu-south-2' | 'eu-west-1' | 'eu-west-2' | 'eu-west-3' | 'il-central-1' | 'me-central-1' | 'me-south-1' | 'sa-east-1' | 'us-east-1' | 'us-east-2' | 'us-gov-east-1' | 'us-gov-west-1' | 'us-west-1' | 'us-west-2', required — The region of the S3 bucket. See here for all region codes. - `format` union, required — Format of the data output. See here for more details - object - `format_type` 'CSV', required - `flattening` 'No flattening' | 'Root level flattening', required — Whether the input json data should be normalized (flattened) in the output CSV. Please refer to docs for details. - `compression` union — Whether the output files should be compressed. If compression is selected, the output filename will have an extra extension (GZIP: ".csv.gz"). - object - … - object - … - object - `format_type` 'JSONL', required - `flattening` 'No flattening' | 'Root level flattening' — Whether the input json data should be normalized (flattened) in the output JSON Lines. Please refer to docs for details. - `compression` union — Whether the output files should be compressed. If compression is selected, the output filename will have an extra extension (GZIP: ".jsonl.gz"). - object - … - object - … - object - `format_type` 'Avro', required - `compression_codec` union, required — The compression algorithm used to compress data. Default to no compression. - object - … - object - … - object - … - object - … - object - … - object - … - object - `format_type` 'Parquet', required - `compression_codec` 'UNCOMPRESSED' | 'SNAPPY' | 'GZIP' | 'LZO' | 'BROTLI' | 'LZ4' | 'ZSTD' — The compression algorithm used to compress data pages. - `block_size_mb` integer — This is the size of a row group being buffered in memory. It limits the memory usage when writing. Larger values will improve the IO when reading, but consume more memory when writing. Default: 128 MB. - `max_padding_size_mb` integer — Maximum size allowed as padding to align row groups. This is also the minimum size of a row group. Default: 8 MB. - `page_size_kb` integer — The page size is for compression. A block is composed of pages. A page is the smallest unit that must be read fully to access a single record. If this value is too small, the compression will deteriorate. Default: 1024 KB. - `dictionary_page_size_kb` integer — There is one dictionary page per column per row group when dictionary encoding is used. The dictionary page size works like the page size but for dictionary. Default: 1024 KB. - `dictionary_encoding` boolean — Default: true. - `s3_endpoint` string — Your S3 endpoint url. Read more here - `s3_path_format` string — Format string on how data will be organized inside the S3 bucket directory. Read more here - `file_name_pattern` string — The pattern allows you to set the file-name format for the S3 staging file(s) - `destinationType` 's3', required - DestinationS3Glue - `access_key_id` string — The access key ID to access the S3 bucket. Airbyte requires Read and Write permissions to the given bucket. Read more here. - `secret_access_key` string — The corresponding secret to the access key ID. Read more here - `s3_bucket_name` string, required — The name of the S3 bucket. Read more here. - `s3_bucket_path` string, required — Directory under the S3 bucket where data will be written. Read more here - `s3_bucket_region` '' | 'af-south-1' | 'ap-east-1' | 'ap-northeast-1' | 'ap-northeast-2' | 'ap-northeast-3' | 'ap-south-1' | 'ap-south-2' | 'ap-southeast-1' | 'ap-southeast-2' | 'ap-southeast-3' | 'ap-southeast-4' | 'ca-central-1' | 'ca-west-1' | 'cn-north-1' | 'cn-northwest-1' | 'eu-central-1' | 'eu-central-2' | 'eu-north-1' | 'eu-south-1' | 'eu-south-2' | 'eu-west-1' | 'eu-west-2' | 'eu-west-3' | 'il-central-1' | 'me-central-1' | 'me-south-1' | 'sa-east-1' | 'us-east-1' | 'us-east-2' | 'us-gov-east-1' | 'us-gov-west-1' | 'us-west-1' | 'us-west-2', required — The region of the S3 bucket. See here for all region codes. - `format` object, required — Format of the data output. See here for more details - `format_type` 'JSONL', required - `compression` union — Whether the output files should be compressed. If compression is selected, the output filename will have an extra extension (GZIP: ".jsonl.gz"). - object - `compression_type` 'No Compression' - object - `compression_type` 'GZIP' - `flattening` 'No flattening' | 'Root level flattening' — Whether the input json data should be normalized (flattened) in the output JSON Lines. Please refer to docs for details. - `s3_endpoint` string — Your S3 endpoint url. Read more here - `s3_path_format` string — Format string on how data will be organized inside the S3 bucket directory. Read more here - `file_name_pattern` string — The pattern allows you to set the file-name format for the S3 staging file(s) - `glue_database` string, required — Name of the glue database for creating the tables, leave blank if no integration - `glue_serialization_library` 'org.openx.data.jsonserde.JsonSerDe' | 'org.apache.hive.hcatalog.data.JsonSerDe', required — The library that your query engine will use for reading and writing data in your lake. - `destinationType` 's3-glue', required - DestinationSftpJson - `host` string, required — Hostname of the SFTP server. - `port` integer — Port of the SFTP server. - `username` string, required — Username to use to access the SFTP server. - `password` string, required — Password associated with the username. - `destination_path` string, required — Path to the directory where json files will be written. - `destinationType` 'sftp-json', required - DestinationSnowflake - `host` string, required — Enter your Snowflake account's locator (in the format ...snowflakecomputing.com) - `role` string, required — Enter the role that you want to use to access Snowflake - `warehouse` string, required — Enter the name of the warehouse that you want to sync data into - `database` string, required — Enter the name of the database you want to sync data into - `schema` string, required — Enter the name of the default schema - `username` string, required — Enter the name of the user you want to use to access the database - `credentials` union - object - `auth_type` 'Key Pair Authentication' - `private_key` string, required — RSA Private key to use for Snowflake connection. See the docs for more information on how to obtain this key. - `private_key_password` string — Passphrase for private key - object - `auth_type` 'Username and Password' - `password` string, required — Enter the password associated with the username. - object - `auth_type` 'OAuth2.0' - `client_id` string — Enter your application's Client ID - `client_secret` string — Enter your application's Client secret - `access_token` string, required — Enter you application's Access Token - `refresh_token` string, required — Enter your application's Refresh Token - `jdbc_url_params` string — Enter the additional properties to pass to the JDBC URL string when connecting to the database (formatted as key=value pairs separated by the symbol &). Example: key1=value1&key2=value2&key3=value3 - `raw_data_schema` string — The schema to write raw tables into (default: airbyte_internal) - `disable_type_dedupe` boolean — Disable Writing Final Tables. WARNING! The data format in _airbyte_data is likely stable but there are no guarantees that other metadata columns will remain the same in future versions - `enable_incremental_final_table_updates` boolean — When enabled your data will load into your final tables incrementally while your data is still being synced. When Disabled (the default), your data loads into your final tables once at the end of a sync. Note that this option only applies if you elect to create Final tables - `destinationType` 'snowflake', required - DestinationTeradata - `host` string, required — Hostname of the database. - `username` string, required — Username to use to access the database. - `password` string — Password associated with the username. - `schema` string — The default schema tables are written to if the source does not specify a namespace. The usual value for this field is "public". - `ssl` boolean — Encrypt data using SSL. When activating SSL, please select one of the connection modes. - `ssl_mode` union — SSL connection modes. disable - Chose this mode to disable encryption of communication between Airbyte and destination database allow - Chose this mode to enable encryption only when required by the destination database prefer - Chose this mode to allow unencrypted connection only if the destination database does not support encryption require - Chose this mode to always require encryption. If the destination database server does not support encryption, connection will fail verify-ca - Chose this mode to always require encryption and to verify that the destination database server has a valid SSL certificate verify-full - This is the most secure mode. Chose this mode to always require encryption and to verify the identity of the destination database server See more information - in the docs. - object — Disable SSL. - `mode` 'disable', required - object — Allow SSL mode. - `mode` 'allow', required - object — Prefer SSL mode. - `mode` 'prefer', required - object — Require SSL mode. - `mode` 'require', required - object — Verify-ca SSL mode. - `mode` 'verify-ca', required - `ssl_ca_certificate` string, required — Specifies the file name of a PEM file that contains Certificate Authority (CA) certificates for use with SSLMODE=verify-ca. See more information - in the docs. - object — Verify-full SSL mode. - `mode` 'verify-full', required - `ssl_ca_certificate` string, required — Specifies the file name of a PEM file that contains Certificate Authority (CA) certificates for use with SSLMODE=verify-full. See more information - in the docs. - `jdbc_url_params` string — Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (example: key1=value1&key2=value2&key3=value3). - `destinationType` 'teradata', required - DestinationTimeplus - `endpoint` string, required — Timeplus workspace endpoint - `apikey` string, required — Personal API key - `destinationType` 'timeplus', required - DestinationTypesense - `api_key` string, required — Typesense API Key - `host` string, required — Hostname of the Typesense instance without protocol. - `port` string — Port of the Typesense instance. Ex: 8108, 80, 443. Default is 443 - `protocol` string — Protocol of the Typesense instance. Ex: http or https. Default is https - `batch_size` integer — How many documents should be imported together. Default 1000 - `destinationType` 'typesense', required - DestinationVectara — Configuration to connect to the Vectara instance - `oauth2` object, required — OAuth2.0 credentials used to authenticate admin actions (creating/deleting corpora) - `client_id` string, required — OAuth2.0 client id - `client_secret` string, required — OAuth2.0 client secret - `customer_id` string, required — Your customer id as it is in the authenticaion url - `corpus_name` string, required — The Name of Corpus to load data into - `parallelize` boolean — Parallelize indexing into Vectara with multiple threads - `text_fields` string[] — List of fields in the record that should be in the section of the document. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered text fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. - `title_field` string — A field that will be used to populate the `title` of each document. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered text fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. - `metadata_fields` string[] — List of fields in the record that should be stored as metadata. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered metadata fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. When specifying nested paths, all matching values are flattened into an array set to a field named by the path. - `destinationType` 'vectara', required - DestinationVertica - `host` string, required — Hostname of the database. - `port` integer, required — Port of the database. - `database` string, required — Name of the database. - `username` string, required — Username to use to access the database. - `password` string — Password associated with the username. - `jdbc_url_params` string — Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (example: key1=value1&key2=value2&key3=value3). - `schema` string, required — Schema for vertica destination - `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use. - object - `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database - object - `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host. - `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa ) - object - `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host - `tunnel_user_password` string, required — OS-level password for logging into the jump server host - `destinationType` 'vertica', required - DestinationWeaviate — The configuration model for the Vector DB based destinations. This model is used to generate the UI for the destination configuration, as well as to provide type safety for the configuration passed to the destination. The configuration model is composed of four parts: * Processing configuration * Embedding configuration * Indexing configuration * Advanced configuration Processing, embedding and advanced configuration are provided by this base class, while the indexing configuration is provided by the destination connector in the sub class. - `embedding` union, required — Embedding configuration - object — Do not calculate and pass embeddings to Weaviate. Suitable for clusters with configured vectorizers to calculate embeddings within Weaviate or for classes that should only support regular text search. - `mode` 'no_embedding', required - object — Use the Azure-hosted OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `mode` 'azure_openai', required - `openai_key` string, required — The API key for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `api_base` string, required — The base URL for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `deployment` string, required — The deployment for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - object — Use the OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `mode` 'openai', required - `openai_key` string, required - object — Use the Cohere API to embed text. - `mode` 'cohere', required - `cohere_key` string, required - object — Use a field in the record as the embedding. This is useful if you already have an embedding for your data and want to store it in the vector store. - `mode` 'from_field', required - `field_name` string, required — Name of the field in the record that contains the embedding - `dimensions` integer, required — The number of dimensions the embedding model is generating - object — Use a fake embedding made out of random vectors with 1536 embedding dimensions. This is useful for testing the data pipeline without incurring any costs. - `mode` 'fake', required - object — Use a service that's compatible with the OpenAI API to embed text. - `mode` 'openai_compatible', required - `api_key` string - `base_url` string, required — The base URL for your OpenAI-compatible service - `model_name` string — The name of the model to use for embedding - `dimensions` integer, required — The number of dimensions the embedding model is generating - `processing` object, required - `chunk_size` integer, required — Size of chunks in tokens to store in vector store (make sure it is not too big for the context if your LLM) - `chunk_overlap` integer — Size of overlap between chunks in tokens to store in vector store to better capture relevant context - `text_fields` string[] — List of fields in the record that should be used to calculate the embedding. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered text fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. - `metadata_fields` string[] — List of fields in the record that should be stored as metadata. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered metadata fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. When specifying nested paths, all matching values are flattened into an array set to a field named by the path. - `text_splitter` union — Split text fields into chunks based on the specified method. - object — Split the text by the list of separators until the chunk size is reached, using the earlier mentioned separators where possible. This is useful for splitting text fields by paragraphs, sentences, words, etc. - `mode` 'separator', required - `separators` string[] — List of separator strings to split text fields by. The separator itself needs to be wrapped in double quotes, e.g. to split by the dot character, use ".". To split by a newline, use "\n". - `keep_separator` boolean — Whether to keep the separator in the resulting chunks - object — Split the text by Markdown headers down to the specified header level. If the chunk size fits multiple sections, they will be combined into a single chunk. - `mode` 'markdown', required - `split_level` integer — Level of markdown headers to split text fields by. Headings down to the specified level will be used as split points - object — Split the text by suitable delimiters based on the programming language. This is useful for splitting code into chunks. - `mode` 'code', required - `language` 'cpp' | 'go' | 'java' | 'js' | 'php' | 'proto' | 'python' | 'rst' | 'ruby' | 'rust' | 'scala' | 'swift' | 'markdown' | 'latex' | 'html' | 'sol', required — Split code in suitable places based on the programming language - `field_name_mappings` object[] — List of fields to rename. Not applicable for nested fields, but can be used to rename fields already flattened via dot notation. - `from_field` string, required — The field name in the source - `to_field` string, required — The field name to use in the destination - `omit_raw_text` boolean — Do not store the text that gets embedded along with the vector and the metadata in the destination. If set to true, only the vector and the metadata will be stored - in this case raw text for LLM use cases needs to be retrieved from another source. - `indexing` object, required — Indexing configuration - `host` string, required — The public endpoint of the Weaviate cluster. - `auth` union, required — Authentication method - object — Authenticate using an API token (suitable for Weaviate Cloud) - `mode` 'token', required - `token` string, required — API Token for the Weaviate instance - object — Authenticate using username and password (suitable for self-managed Weaviate clusters) - `mode` 'username_password', required - `username` string, required — Username for the Weaviate cluster - `password` string, required — Password for the Weaviate cluster - object — Do not authenticate (suitable for locally running test clusters, do not use for clusters with public IP addresses) - `mode` 'no_auth', required - `batch_size` integer — The number of records to send to Weaviate in each batch - `text_field` string — The field in the object that contains the embedded text - `tenant_id` string — The tenant ID to use for multi tenancy - `default_vectorizer` 'none' | 'text2vec-cohere' | 'text2vec-huggingface' | 'text2vec-openai' | 'text2vec-palm' | 'text2vec-contextionary' | 'text2vec-transformers' | 'text2vec-gpt4all' — The vectorizer to use if new classes need to be created - `additional_headers` object[] — Additional HTTP headers to send with every request. - `header_key` string, required - `value` string, required - `destinationType` 'weaviate', required - DestinationXata - `api_key` string, required — API Key to connect. - `db_url` string, required — URL pointing to your workspace. - `destinationType` 'xata', required ## Other responses - `400` — Invalid data - `403` — Not allowed - `404` — Not found --- [API](https://skmtc.dev/airbyte/apis/applications.md) · [All operations](https://skmtc.dev/airbyte/apis/applications/llms.txt) · [OpenAPI document](https://skmtc-service-production.skmtc.workers.dev/v1/apis/airbyte/applications/revisions/561bf5551285/schema)