KEkeboola
Buckets

Bucket Refresh

EXPERIMENTAL

Refresh external bucket. For Snowflake and BigQuery backends only. If the database object (DB, schema...) isn't available anymore (was deleted or revoked), the bucket in Storage will be deleted.

By default only tables whose backend LAST_ALTERED is newer than the bucket's stored lastChangeDate are refreshed. Use force=1 to ignore the stored date and refresh all tables and views (useful to recover from a stale Snowflake LAST_ALTERED on a data share), or pass lastChangeDate to override the comparison date. force takes precedence over lastChangeDate.

A regular (not external, not linked) Snowflake bucket in the default branch can be refreshed too. The refresh registers the tables of the bucket schema that Storage does not know yet - for example a table restored with UNDROP TABLE or created outside Keboola - with their columns, types, primary key and comments as descriptions, and transfers their ownership to the project role. Tables Storage already knows are never changed or removed, import staging tables (__temp_ prefix) and objects other than tables are skipped, and an empty table younger than about 15 minutes is left for a later refresh because it may belong to a table create still in progress. force and lastChangeDate do not apply.

Initial response returns job details JobResponse. The result of the asynchronous job can be fetched via /v2/storage/jobs/{id}. When the job completes, final result is RefreshJobResponse.

Parameters of job will be RefreshJobParams.

put/v2/storage/buckets/{bucketId}/refresh

Request

Path parameters

bucketIdstring required

Bucket ID.

Query parameters

forceboolean

When true, ignore the stored lastChangeDate and refresh all tables and views. Takes precedence over lastChangeDate.

lastChangeDatestring nullable

Override of the lastChangeDate used to detect altered tables. Can be a unix timestamp or any date accepted by strtotime. Examples: -2 days, 1360138863, 2013-02-12T15:19:21+00:00. Ignored when force is true.

Response

Job accepted; returns job details.

idstring required

Job ID.

statusstring required

Current status of the job.

urlstring required

API detail URL for the job.

tableIdstring nullable required

ID of the table affected by the job.

operationNamestring required

Type of operation performed.

operationParamsobject required

Parameters for the operation.

createdTimestring date-time required

Job creation timestamp.

startTimestring date-time nullable required

Job start timestamp.

endTimestring date-time nullable required

Job end timestamp.

runIdstring nullable required

Run ID for the job.

Example response

{
  "id": "22077337",
  "status": "waiting",
  "url": "https://connection.keboola.com/v2/storage/jobs/22077337",
  "tableId": null,
  "operationName": "workspaceLoad",
  "operationParams": {
    "workspaceId": "78423",
    "preserve": false,
    "input": [
      {
        "source": "in.c-application-testing.cashier-data",
        "destination": "my-table"
      }
    ],
    "queue": "main_fast"
  },
  "createdTime": "2017-02-13T16:41:18+0100",
  "startTime": null,
  "endTime": null,
  "runId": null,
  "results": null,
  "creatorToken": {
    "id": "27978",
    "description": "john.doe@keboola.com"
  },
  "actor": {
    "type": "token",
    "id": "27978",
    "name": "john.doe@keboola.com",
    "adminId": 1234
  },
  "metrics": {
    "inCompressed": false,
    "inBytes": 0,
    "inBytesUncompressed": 0,
    "outCompressed": false,
    "outBytes": 0,
    "outBytesUncompressed": 0
  }
}

Changes

No recorded changes to this operation across all 1 revision of this API.