You can create a process for the following:

• Data pipeline brick
• Ruby script
• CloudConnect graph </br> </br>

After you create a process, you can create a schedule for it. For more information about schedules, see Schedules of Data Loading Processes.

For a process for a CloudConnect graph, do the following before making this call:

1. Archive or zip the entire CloudConnect Project that you want to load and name it upload.zip. The ZIP file must be smaller than 5 MB.
2. Upload the archive or zip file to your GoodData user-specific storage.

If you are using Automated Data Distribution v1 (ADD v1), which can load data from the GoodData Data Warehouse only, you do not have to create a process for it. The process is automatically created after you configure the output stage parameters for the workspace where you want to load data from Data Warehouse. The process is created under the name of 'Automated Data Distribution'.

If you are creating a process for a data pipeline brick, you can include Data Sources so that the process can use parameters from those Data Sources. For more information, see Reuse Parameters in Multiple Bricks.

In the request, use the type parameter to specify the type of the process that you want to create: </br>

• GRAPH for a CloudConnect graph
• RUBY for a Ruby script (for more information about Ruby process deployment, see Deploy a Data Loading Process from a Ruby Script)
• ETL for Automated Data Distribution v2 (ADD v2) or a data pipeline brick. For this process type, also define whether you want to use ADD v2 or the brick in the process>component>name parameter:
  • gdc-data-distribution for Automated Data Distribution v2
  • gdc-etl-csv-downloader for CSV Downloader
  • gdc-etl-ga-downloader for Google Analytics Downloader
  • gdc-etl-gsheets-downloader for Google Sheets Downloader
  • gdc-etl-salesforce-downloader for Salesforce Downloader
  • gdc-etl-sql-downloader for SQL Downloader
  • gdc-etl-ads-integrator for ADS Integrator
  • gdc-etl-sql-executor for SQL Executor
  • gdc-etl-schedule-executor for Schedule Executor
  • gdc-brick-object-rename for Object Renaming Utility
  • gdc-brick-ldm-custom-fields for Custom Field Creator
  • gdc-brick-date-dimension-loader for Date Dimension Loader

</br>

This is a sample request for creating a process of the GRAPH type.

{
    "process" : {
        "type": "GRAPH",
        "name": "{process_name}"
        "path": "{path/to/archive.zip}"
    }
}

This is a sample request for creating a process of the ETL type (Automated Data Distribution v2).

{
    "process": {
        "type": "ETL",
        "name": "ADD V2",
        "component": {
            "name": "gdc-data-distribution",
            "version": "1",
            "config": {
                "dataDistribution": {
                    "dataSource": "{data_source_id}",
                    "target": {
                        "segment": {
                            "uri": "/gdc/domains/{domain}/dataproducts/{data_product}/segments/{segment}"
                        }
                    }
                }
            }
        }
    }
}

This is a sample request for creating a process of the ETL type (data pipeline bricks).

{
    "process": {
        "type": "ETL",
        "name": "{process_name}",
        "component": {
            "name": "{brick}",
            "version": "1",
            "configLocation": {
                "s3": {
                    "path": "s3://{bds_bucket}/{path/to/configuration}/",
                    "accessKey": "{s3_access_key}",
                    "secretKey": "{s3_secret_key}",
                    "region": "us-east-1",
                    "serverSideEncryption": true
                }
            }
        },
        "dataSources": [
            {
                "id": "{data_source_01_id}"
            },
            {
                "id": "{data_source_02_id}"
            }
        ]
    }
}

This is a sample request for creating a process of the ETL type (data pipeline bricks) using parameters from an S3 Data Source.

{
    "process": {
        "type": "ETL",
        "name": "{process_name}",
        "component": {
            "name": "{brick}",
            "version": "1",
            "configLocation": {
                "dataSourceConfig": {
                        "path": "{subfolder_path}",
                        "id": "{s3_data_source_id}"
                }
            }
        },
        "dataSources": [
            {
                "id": "{data_source_01_id}"
            },
            {
                "id": "{data_source_02_id}"
            }
        ]
    }
}

This is a sample response for a created process of the GRAPH type. executables lists executables (CloudConnect graphs / Ruby scripts / Groovy scripts) in the uploaded zip file.

{
    "process" : {
        "type": "GRAPH",
        "name": "{process_name}",
        "graphs": [
            "{graph_1.grf}"
        ],
        "executables": [
            "graph/{graph_1}.grf",
            "graph/{graph_2}.grf"
        ],
        "ownerLogin": "{owner_email_address}",
        "links": {
            "self": "/gdc/projects/{project_id}/dataload/processes/{process_id}",
            "executions": "/gdc/projects/{project_id}/dataload/processes/{process_id}/executions",
            "source": "/gdc/projects/{project_id}/dataload/processes/{process_id}/source"
        }
    }
}

This is a sample response for a created process of the ETL type (Automated Data Distribution v2).

{
    "process": {
        "type": "ETL",
        "name": "Automated Data Distribution",
        "lastDeployed": "YYYY-MM-DD'T'HH:MM:SS.SSS'Z'",
        "ownerLogin": "{owner_email_address}",
        "links": {
            "self": "/gdc/projects/{project_id}/dataload/processes/{process_id}",
            "executions": "/gdc/projects/{project_id}/dataload/processes/{process_id}/executions",
            "source": "/gdc/projects/{project_id}/dataload/processes/{process_id}/source"
        },
        "component": {
            "name": "gdc-data-distribution",
            "version": "1",
            "config": {
                "dataDistribution": {
                    "dataSource": "{data_source_id}",
                    "target": {
                        "segment": {
                            "uri": "/gdc/domains/{domain}/dataproducts/{data_product}/segments/{segment}"
                        }
                    }
                }
            }
        }
    }
}

This is a sample response for a created process of the ETL type (data pipeline bricks).

{
    "process": {
        "type": "ETL",
        "name": "{process_name}",
        "lastDeployed": "YYYY-MM-DD'T'HH:MM:SS.SSS'Z'",
        "ownerLogin": "{owner_email_address}",
        "links": {
            "self": "/gdc/projects/{project_id}/dataload/processes/{process_id}",
            "executions": "/gdc/projects/{project_id}/dataload/processes/{process_id}/executions",
            "source": "/gdc/projects/{project_id}/dataload/processes/{process_id}/source"
        },
        "component": {
            "name": "{brick}",
            "version": "1",
            "configLocation": {
                "s3": {
                    "path": "s3://{bds_bucket}/{path/to/configuration}/",
                    "accessKey": "{s3_access_key}",
                    "region": "us-east-1",
                    "serverSideEncryption": true
                }
            },
            "dataSources": [
                {
                    "id": "{data_source_01_id}"
                },
                {
                    "id": "{data_source_02_id}"
                }
            ]
        }
    }
}

This is a sample response for a created process of the ETL type (data pipeline bricks) using parameters from an S3 Data Source.

{
    "process": {
        "type": "ETL",
        "name": "{process_name}",
        "lastDeployed": "YYYY-MM-DD'T'HH:MM:SS.SSS'Z'",
        "ownerLogin": "{owner_email_address}",
        "links": {
            "self": "/gdc/projects/{project_id}/dataload/processes/{process_id}",
            "executions": "/gdc/projects/{project_id}/dataload/processes/{process_id}/executions",
            "source": "/gdc/projects/{project_id}/dataload/processes/{process_id}/source"
        },
        "component": {
            "name": "{brick}",
            "version": "1",
            "configLocation": {
                "dataSourceConfig": {
                    "path": "{subfolder_path}",
                    "id": "{s3_data_source_id}"
                }
            },
            "dataSources": [
                {
                    "id": "{data_source_01_id}",
                },
                {
                    "id": "{data_source_02_id}"
                }
            ]
        }
    }
}

{
  "process": {
    "alias": "some_transformation_123",
    "type": "GRAPH",
    "name": "some_transformation",
    "component": {},
    "dataSources": [
      {
        "id": "data_source_id"
      }
    ],
    "path": "{path}",
    "ownerLogin": "{ownerLogin}",
    "graphs": [],
    "executables": [
      "graph/downloader.grf",
      "graph/gd_load.grf",
      "graph/jsons_parser.grf",
      "graph/run.grf",
      "graph/transform.grf"
    ]
  }
}

{
  "process": {
    "alias": "some_transformation_123",
    "type": "GRAPH",
    "name": "some_transformation",
    "component": {},
    "dataSources": [
      {
        "id": "data_source_id"
      }
    ],
    "ownerLogin": "{ownerLogin}",
    "graphs": [],
    "executables": [
      "graph/downloader.grf",
      "graph/gd_load.grf",
      "graph/jsons_parser.grf",
      "graph/run.grf",
      "graph/transform.grf"
    ],
    "links": {
      "self": "/gdc/projects/{project_id}/dataload/processes/{process_id}",
      "executions": "/gdc/projects/{project_id}/dataload/processes/{process_id}/executions"
    }
  }
}