--- title: "Create a data loading process" method: POST path: "/gdc/projects/{projectId}/dataload/processes" tags: ["dataloading"] --- # Create a data loading process `POST /gdc/projects/{projectId}/dataload/processes` ## Request - Base URL: `https://secure.gooddata.com` - URL: `https://secure.gooddata.com/gdc/projects/{projectId}/dataload/processes` - Auth: none declared You can create a process for the following: - [Data pipeline brick](https://help.gooddata.com/display/doc/Data+Preparation+and+Distribution) - Ruby script - CloudConnect graph

After you create a process, you can create a schedule for it. For more information about schedules, see [Schedules of Data Loading Processes](#tag/schedules). For a process for a **CloudConnect graph**, do the following before making this call: 1. Archive or zip the entire CloudConnect Project that you want to load and name it **upload.zip**. The ZIP file must be smaller than 5 MB. 2. Upload the archive or zip file to your GoodData [user-specific storage](https://help.gooddata.com/display/doc/User+Specific+Data+Storage). If you are using [**Automated Data Distribution v1 (ADD v1)**](https://help.gooddata.com/display/doc/Automated+Data+Distribution), which can load data from the GoodData Data Warehouse only, you do not have to create a process for it. The process is automatically created after you configure the output stage parameters for the workspace where you want to load data from Data Warehouse. The process is created under the name of 'Automated Data Distribution'. If you are creating a process for a **data pipeline brick**, you can include Data Sources so that the process can use parameters from those Data Sources. For more information, see [Reuse Parameters in Multiple Bricks](https://help.gooddata.com/display/doc/Reuse+Parameters+in+Multiple+Bricks). In the request, use the `type` parameter to specify the type of the process that you want to create:
- `GRAPH` for a CloudConnect graph - `RUBY` for a Ruby script (for more information about Ruby process deployment, see [Deploy a Data Loading Process from a Ruby Script](https://help.gooddata.com/pages/viewpage.action?pageId=86796352)) - `ETL` for Automated Data Distribution v2 (ADD v2) or a [data pipeline brick](https://help.gooddata.com/pages/viewpage.action?pageId=86796083). For this process type, also define whether you want to use ADD v2 or the brick in the `process`>`component`>`name` parameter: - `gdc-data-distribution` for Automated Data Distribution v2 - `gdc-etl-csv-downloader` for CSV Downloader - `gdc-etl-ga-downloader` for Google Analytics Downloader - `gdc-etl-gsheets-downloader` for Google Sheets Downloader - `gdc-etl-salesforce-downloader` for Salesforce Downloader - `gdc-etl-sql-downloader` for SQL Downloader - `gdc-etl-ads-integrator` for ADS Integrator - `gdc-etl-sql-executor` for SQL Executor - `gdc-etl-schedule-executor` for Schedule Executor - `gdc-brick-object-rename` for Object Renaming Utility - `gdc-brick-ldm-custom-fields` for Custom Field Creator - `gdc-brick-date-dimension-loader` for Date Dimension Loader
This is a sample request for creating a process of the GRAPH type. ``` { "process" : { "type": "GRAPH", "name": "{process_name}" "path": "{path/to/archive.zip}" } } ``` This is a sample request for creating a process of the ETL type (Automated Data Distribution v2). ``` { "process": { "type": "ETL", "name": "ADD V2", "component": { "name": "gdc-data-distribution", "version": "1", "config": { "dataDistribution": { "dataSource": "{data_source_id}", "target": { "segment": { "uri": "/gdc/domains/{domain}/dataproducts/{data_product}/segments/{segment}" } } } } } } } ``` This is a sample request for creating a process of the ETL type (data pipeline bricks). ``` { "process": { "type": "ETL", "name": "{process_name}", "component": { "name": "{brick}", "version": "1", "configLocation": { "s3": { "path": "s3://{bds_bucket}/{path/to/configuration}/", "accessKey": "{s3_access_key}", "secretKey": "{s3_secret_key}", "region": "us-east-1", "serverSideEncryption": true } } }, "dataSources": [ { "id": "{data_source_01_id}" }, { "id": "{data_source_02_id}" } ] } } ``` This is a sample request for creating a process of the ETL type (data pipeline bricks) using parameters from an S3 Data Source. ``` { "process": { "type": "ETL", "name": "{process_name}", "component": { "name": "{brick}", "version": "1", "configLocation": { "dataSourceConfig": { "path": "{subfolder_path}", "id": "{s3_data_source_id}" } } }, "dataSources": [ { "id": "{data_source_01_id}" }, { "id": "{data_source_02_id}" } ] } } ``` This is a sample response for a created process of the GRAPH type. `executables` lists executables (CloudConnect graphs / Ruby scripts / Groovy scripts) in the uploaded zip file. ``` { "process" : { "type": "GRAPH", "name": "{process_name}", "graphs": [ "{graph_1.grf}" ], "executables": [ "graph/{graph_1}.grf", "graph/{graph_2}.grf" ], "ownerLogin": "{owner_email_address}", "links": { "self": "/gdc/projects/{project_id}/dataload/processes/{process_id}", "executions": "/gdc/projects/{project_id}/dataload/processes/{process_id}/executions", "source": "/gdc/projects/{project_id}/dataload/processes/{process_id}/source" } } } ``` This is a sample response for a created process of the ETL type (Automated Data Distribution v2). ``` { "process": { "type": "ETL", "name": "Automated Data Distribution", "lastDeployed": "YYYY-MM-DD'T'HH:MM:SS.SSS'Z'", "ownerLogin": "{owner_email_address}", "links": { "self": "/gdc/projects/{project_id}/dataload/processes/{process_id}", "executions": "/gdc/projects/{project_id}/dataload/processes/{process_id}/executions", "source": "/gdc/projects/{project_id}/dataload/processes/{process_id}/source" }, "component": { "name": "gdc-data-distribution", "version": "1", "config": { "dataDistribution": { "dataSource": "{data_source_id}", "target": { "segment": { "uri": "/gdc/domains/{domain}/dataproducts/{data_product}/segments/{segment}" } } } } } } } ``` This is a sample response for a created process of the ETL type (data pipeline bricks). ``` { "process": { "type": "ETL", "name": "{process_name}", "lastDeployed": "YYYY-MM-DD'T'HH:MM:SS.SSS'Z'", "ownerLogin": "{owner_email_address}", "links": { "self": "/gdc/projects/{project_id}/dataload/processes/{process_id}", "executions": "/gdc/projects/{project_id}/dataload/processes/{process_id}/executions", "source": "/gdc/projects/{project_id}/dataload/processes/{process_id}/source" }, "component": { "name": "{brick}", "version": "1", "configLocation": { "s3": { "path": "s3://{bds_bucket}/{path/to/configuration}/", "accessKey": "{s3_access_key}", "region": "us-east-1", "serverSideEncryption": true } }, "dataSources": [ { "id": "{data_source_01_id}" }, { "id": "{data_source_02_id}" } ] } } } ``` This is a sample response for a created process of the ETL type (data pipeline bricks) using parameters from an S3 Data Source. ``` { "process": { "type": "ETL", "name": "{process_name}", "lastDeployed": "YYYY-MM-DD'T'HH:MM:SS.SSS'Z'", "ownerLogin": "{owner_email_address}", "links": { "self": "/gdc/projects/{project_id}/dataload/processes/{process_id}", "executions": "/gdc/projects/{project_id}/dataload/processes/{process_id}/executions", "source": "/gdc/projects/{project_id}/dataload/processes/{process_id}/source" }, "component": { "name": "{brick}", "version": "1", "configLocation": { "dataSourceConfig": { "path": "{subfolder_path}", "id": "{s3_data_source_id}" } }, "dataSources": [ { "id": "{data_source_01_id}", }, { "id": "{data_source_02_id}" } ] } } } ``` ## Request body - Process1 - `process` ProcessRequest, required - `alias` string, nullable: The process alias - `type` string: The type of the process: DATALOAD, RUBY, GRAPH or ETL. Default is GRAPH - `name` string, required: The process name - `component` Component1 - `name` string, required - `version` string, required - `earlyAccessVersion` string, nullable - `configLocation` union - S3Config1 - `s3` S3ConfigRequest, required - `path` string, required - `accessKey` string, required - `region` string - `serverSideEncryption` true | false - `secretKey` string, nullable - DataSourceConfig - `dataSourceConfig` object, required - `id` string, required - `path` string - `config` DataDistributionConfig - `dataDistribution` object, required - `dataSource` string, required - `path` string - `target` SegmentTarget - `segment` object, required - … - `dataSources` DataSourceItem[] - `id` string, required - `path` string - `ownerLogin` string, nullable - `graphs` string[]: List of graphs contained in the ZIP file, DEPRECATED - use executables instead - `executables` string[]: The list of executables (CloudConnect graphs / Ruby scripts) in the uploaded zip file ## Response `201` OK - Process - `process` ProcessResponse, required - `alias` string, nullable: The process alias - `type` string: The type of the process: DATALOAD, RUBY, GRAPH or ETL. Default is GRAPH - `name` string, required: The process name - `component` Component - `name` string, required - `version` string, required - `earlyAccessVersion` string, nullable - `configLocation` union - S3Config - `s3` S3ConfigResponse, required - `path` string, required - `accessKey` string, required - `region` string - `serverSideEncryption` true | false - `secretKey` string, nullable - DataSourceConfig - `dataSourceConfig` object, required - `id` string, required - `path` string - `config` DataDistributionConfig - `dataDistribution` object, required - `dataSource` string, required - `path` string - `target` SegmentTarget - `segment` object, required - … - `dataSources` DataSourceItem[] - `id` string, required - `ownerLogin` string, nullable - `graphs` string[]: list of graphs contained in the ZIP file, DEPRECATED - use executables instead - `executables` string[]: list of executables - graphs, scripts etc. - `path` string - `links` object - `self` string, required - `executions` string, required ## Other responses - `202`: Accepted - in case of git process deployment the async task with poll link --- [API](https://skmtc.dev/gooddata/apis/gooddata-api-documentation.md) - [All operations](https://skmtc.dev/gooddata/apis/gooddata-api-documentation/llms.txt) - [OpenAPI document](https://skmtc.dev/gooddata/apis/gooddata-api-documentation/revisions/e5dc0ee6ab00?raw)