Execute a schedule
Places the specified schedule in the execution queue. The schedule will run as soon as possible. </br> </br> Depending on whether you are executing an Automated Data Distribution (ADD) schedule or a regular (non ADD) schedule, you can specify various execution parameters in the request body. For more information about Automated Data Distribution (ADD), see Automated Data Distribution. </br> </br> Parameters for all schedules: </br>
- retry (Boolean, optional) - If true, applies the reschedule property if the schedule has it set (see Configure Automatic Retry of a Failed Data Loading Process and Schedule a data loading process ). When not set, defaults to false. </br> NOTE: retry is applied only when the schedule has the retry delay set (that is, the reschedule property is set). If the schedule does not have it set, retry is ignored, and the retry delay is not applied for the schedule. </br> Example: You want to execute a schedule with reschedule set to 30 (Restart every 30 minutes until success (or 30th consecutive failure)). If you set retry to true in the request body, then in case of execution failure the schedule will be rerun again in 30 minutes.If retry is set to false or not set at all, the schedule will not be rerun.</br> </br>
Parameters specific to ADD schedules only:</br>
- queryParallelism (number, optional) specifies how many queries will be executed in parallel. </br>
NOTE: This parameter is in beta release. Use it with caution. </br>
The parameter can be set to 1 (datasets are uploaded one by one sequentially) or 2 (two datasets are uploaded in parallel, which speeds up the uploading process).
- If the parameter is not explicitly set for schedule execution, then the value of this parameter in the schedule itself (if set) is used during execution.
- If you explicitly set this parameter for schedule execution, its value will override whatever value this parameter has been set to in the schedule itself.
- GDC_DATALOAD_SINGLE_RUN_LOAD_MODE (string, optional) defines load mode for the executed schedule. Can be DEFAULT, FULL, INCREMENTAL, CUSTOM, or UNRESTRICTED. If not set, defaults to DEFAULT.
- DEFAULT mode selects load mode according to the definition of the dataset table within the ADD output stage. If the dataset table is incremental, the dataset is loaded in incremental mode. If the dataset table is not incremental, the dataset is loaded in full mode. When you set load mode to DEFAULT, no other parameters are required.
- FULL mode loads all data to the datasets. The datasets are completely rewritten. When you set load mode to FULL, no other parameters are required.
- INCREMENTAL mode loads data only after a certain point in time that is defined based on the x__timestamp column in the mapped output stage table. However, if a particular dataset's table is not incremental (the x__timestamp column is not present), it will be excluded from processing, and its data will not be uploaded.
- GDC_DATALOAD_SINGLE_RUN_DATE_FROM and GDC_DATALOAD_SINGLE_RUN_DATE_TO define an interval within which the data should be uploaded to a dataset. Specify either of these parameters, or both.
- CUSTOM mode lets you specify load mode for each dataset to which you want to upload data. For CUSTOM load mode, set the following parameters for each dataset:
- dataset (string, mandatory) defines the dataset identifier.
- uploadMode (string, mandatory) defines load mode for a specific dataset. Can be FULL, INCREMENTAL, or DEFAULT.
- FULL loads a dataset in full mode.
- INCREMENTAL loads a dataset in incremental mode. If the dataset's table is not incremental (the x__timestamp column is not present), it will not be processed, and its data will not be uploaded.
- from and to define an interval within which the data should be uploaded to a dataset. Specify either of these parameters, or both.
- DEFAULT selects load mode (full or incremental) according to the definition of the dataset table within the ADD output stage.
- UNRESTRICTED mode (supported only in ADD v1) lets you specify load mode for each dataset to which you want to upload data and lets you enrich queries for fetching data from the ADD output stage and set ADD system parameters.
You can use this mode when you want to create your own logic for fetching data from the output stage and distributing it to projects, or when you want to manually perform corrective actions in your projects.
For example, if you are uploading data for employees in different countries, you may want to first upload the data for the EU employees only, and then upload the rest of the data.
To exclude deleted records (that is, the records where the x__deleted column contains TRUE) from processing, include the condition parameter in the call request and set it to "x__deleted=FALSE" (see further in this section). For more information,
see Delete Data from Datasets in Automated Data Distribution.<br/>
NOTE: To be able to use UNRESTRICTED mode, you must have access to the Data Warehouse instance that is set in the ADD output stage for the project. <br/>
WARNING: Use UNRESTRICTED mode with caution. Unlike the other modes, UNRESTRICTED mode may lead to data inconsistency in your projects if used with wrong parameters.<br/>
You should use UNRESTRICTED mode only if you are confident in executing the schedule via the API and have a deep understanding of your data.</br>
- GDC_DATALOAD_SKIP_VALIDATE_MAPPING (boolean, optional) specifies whether to skip validation of all the datasets in the logical data model (LDM) against the objects in the output stage when executing the schedule. If not set, defaults to false.<br/> NOTE: This parameter is valid only when the customColumns parameter (see later in this section) is specified for at least one dataset.
- dataset (string, mandatory) defines the dataset identifier.
- uploadMode (string, mandatory) defines load mode for a specific dataset. Can be FULL, INCREMENTAL, or DEFAULT.
- FULL loads a dataset in full mode.
- INCREMENTAL loads a dataset in incremental mode. If the dataset's table is not incremental (the x__timestamp column is not present), it will not be processed, and its data will not be uploaded.
- DEFAULT selects load mode (full or incremental) according to the definition of the dataset table within the ADD output stage.
- condition (string, optional) defines a condition to narrow down the fetch query. When defined, adds an AND operator to the end of SQL fetch query.</br> Example: "a__company=ACME", "a__company=ACME AND x__deleted=FALSE", "d__date BETWEEN '2018-12-01' AND '2018-11-01'" </br>
- includeLSLTSCondition (boolean, optional) specifies whether to automatically add the LSLTS (Last Successfully Loaded Time Stamp) as a condition to the fetch query. If not set, defaults to false.
- customColumns (string, optional) defines columns in the ADD output stage table from which the data should be loaded to custom fields in the corresponding dataset.
When defined, adds these columns to the SELECT section of the fetch query.</br>
Custom fields in a dataset in a project's logical data model (LDM) are fields (such as facts, attributes, and so on) that are added to the dataset and not explicitly mapped
to columns in the corresponding table in the output stage. For example, custom fields can be added to a client project's LDM after the LDM has been propagated from the segment's
master project in an LCM environment.</br>
For successful data load, you must specify where the data to upload to the custom fields comes from. If the data for a custom field is stored in a column whose name does
not conform to the naming conventions or the data should be first extracted from a data object in a table column,
use the customColumns parameter.
- If the name of a column is not according to the naming conventions, provide a name alias for this column that follows the naming conventions.</br> Example: column_custom_price AS f__custom_price
- If the data to load should be extracted from a data object stored in a column, use the appropriate Vertica functions to do so and provide a name alias for the virtual column that follows the naming conventions.</br> Example: If the product_info column of the dataset's table contains the JSON object { "color":"red", "description":"small" } and you need to upload the value red to the custom_color attribute of the dataset, specify the following:</br> MAPLOOKUP(MAPJSONEXTRACTOR(product_info), 'color')::VARCHAR(100) AS a__custom_color</br> TIP: If you set up the customColumns parameter for many datasets or the customColumns parameter contains numerous table columns to load data from, you can use the GDC_DATALOAD_SKIP_VALIDATE_MAPPING parameter (see earlier in this section) to skip the dataset validation against the objects in the output stage and reduce the time needed for the schedule execution. Note that when the GDC_DATALOAD_SKIP_VALIDATE_MAPPING parameter is used, the validation is skipped for all the datasets in the LDM, not only for those with the customColumns parameter specified.
- updateLSLTS (string, optional) overrides the current value of the LSLTS of the dataset and sets it to the specified value. Use the LSLTS format ("2018-12-12 00:00:00:000000"), or set it to AUTO to use the maximum value of the LSLTS from the retrieved data.
- updateCustomUploadState (string, optional) defines a custom upload state (CUS) that is stored when the data upload was successful. If the data upload did not complete for any reason, the CUS is not stored. </br> Example: If you are uploading the data for the EU employees first, you may specify a CUS of EU to be stored in the system. Before you upload another portion of data, you may want to learn whether the first portion of data was uploaded successfully. To do so, use the API for listing all datasets in a project and review the current CUS. If the returned CUS is not set to EU, it means that the first portion of data was not uploaded successfully, and you may want to retry the upload.
- loadDeleteSideTable (boolean, optional) specifies whether to process the delete side table during the data load. If not set, defaults to false.
- GDC_TARGET_PROJECTS (string, optional; supported only in ADD v2 with LCM segments configured) specifies the IDs of the projects to synchronize instead of synchronizing the whole segment. </br> Example: pid123,pid456,pid789 </br>
- dataload_groups_query (string, optional; supported only in ADD v2 with LCM segments configured) lets you load data to a particular set of projects within a segment. For more information, see Schedule a Data Loading Process for a Group of Workspaces.</br> Example: SELECT x__client_id, x__dataload_group, x__priority from load_workspaces
- dataload_groups (string, optional; supported only in ADD v2 with LCM segments configured) specifies the names of the groups containing the projects where you want to load data to. For more information, see Schedule a Data Loading Process for a Group of Workspaces.</br> Example: group 1,group_2
- GDC_DOWNLOAD_PROJECTS_CHUNK_SIZE (number, optional; supported only in ADD v2 with LCM segments configured) specifies how many projects should be retrieved and downloaded from Data Sources in one query. If not specified, defaults to 100. </br>
Response: </br> user_id - The ID (e-mail address) of the user account under which the schedule runs (that is, the user account that deployed the ETL)</br>
This is a sample request for executing a schedule with no parameters.</br>
{
"execution": {}
}
This is a sample request for executing a schedule with the retry parameter set to true.</br>
{
"execution": {
"params": {
"retry": "true"
}
}
}
This is a sample request for executing an ADD schedule in FULL mode.</br>
{
"execution": {
"params": {
"GDC_DATALOAD_SINGLE_RUN_LOAD_MODE": "FULL"
}
}
}
This is a sample request for executing an ADD schedule in INCREMENTAL mode.</br>
{
"execution": {
"params": {
"GDC_DATALOAD_SINGLE_RUN_LOAD_MODE": "INCREMENTAL",
"GDC_DATALOAD_SINGLE_RUN_DATE_FROM": "2016-12-31 14:55:59.000"
}
}
}
This is a sample request for executing an ADD schedule in CUSTOM mode.</br>
{
"execution": {
"params": {
"GDC_DATALOAD_SINGLE_RUN_LOAD_MODE": "CUSTOM",
"GDC_DATALOAD_DATASETS": "[
{
\"dataset\": \"dataset.id_1\",
\"uploadMode\": \"INCREMENTAL\"
},
{
\"dataset\": \"dataset.id_2\",
\"uploadMode\": \"INCREMENTAL\",
\"from\": \"2016-12-31 14:55:59.000\",
\"to\": \"2017-12-31 14:55:59.000\"
},
{
\"dataset\": \"dataset.id_3\",
\"uploadMode\": \"FULL\"
},
{
\"dataset\": \"dataset.id_4\",
\"uploadMode\": \"DEFAULT\"
}
]"
}
}
}
This is a sample request for executing an ADD schedule in UNRESTRICTED mode.</br>
{
"execution": {
"params": {
"GDC_DATALOAD_SINGLE_RUN_LOAD_MODE": "UNRESTRICTED",
"GDC_DATALOAD_SKIP_VALIDATE_MAPPING": "true",
"GDC_DATALOAD_DATASETS": "[
{
\"dataset\": \"dataset.product\",
\"uploadMode\": \"FULL\",
\"customColumns\": \"MAPLOOKUP(MAPJSONEXTRACTOR(product_info), 'color') AS a__color\"
}
{
\"dataset\": \"dataset.orders\",
\"uploadMode\": \"INCREMENTAL\",
\"condition\": \"d__date BETWEEN '2018-12-31' AND '2018-11-30' AND r__merchantid=2\"
},
{
\"dataset\": \"dataset.merchants\",
\"uploadMode\": \"INCREMENTAL\",
\"condition\": \"a__company=ACME\",
\"loadDeleteSideTable\": true,
\"includeLSLTSCondition\": true,
\"updateLSLTS\": \"2018-12-31 00:00:00.000000\",
\"updateCustomUploadState\": \"ACME,MYCOMPANY\"
},
{
\"dataset\": \"dataset.customers\",
\"uploadMode\": \"DEFAULT\"
}
]"
}
}
}
Request
- Base URLhttps://secure.gooddata.com
- Authnone declared
Request body
Response
OK - creates and returns new execution
Example response
{
"execution": {
"createdTime": "2013-11-14T00:00:40.355Z",
"startTime": "2013-11-15T00:00:40.355Z",
"endTime": "2013-11-15T00:00:49.030Z",
"log": "/gdc/projects/{project_id}/dataload/processes/{process_id}/executions/{execution_id}/log",
"proxyHost": "na-apiproxy.com",
"data": "{data}",
"status": "OK",
"error": {},
"links": {
"self": "/gdc/projects/{project_id}/schedules/{schedule_id}/executions/{schedule_execution_id}"
}
}
}Changes
No recorded changes to this operation across all 1 revision of this API.