Pipeline Lifecycle

Recompile Pipeline

Changed on

Required role: write or higher.

Recompile a pipeline with the Feldera runtime version included in the currently installed Feldera platform.

Use this endpoint after upgrading Feldera to rebuild pipelines that were compiled with older platform versions. In most cases, recompilation is not required; pipelines compiled with older versions will continue to run on the upgraded platform.

Situations where recompilation may be necessary:

  • To benefit from the latest bug fixes and performance optimizations.
  • When backward-incompatible changes are introduced in Feldera. In this case, attempting to start a pipeline compiled with an unsupported version will result in an error.

If the pipeline is already compiled with the current platform version, this operation is a no-op.

Note that recompiling the pipeline with a new platform version may change its query plan. If the modified pipeline is started from an existing checkpoint, it may require bootstrapping parts of its state from scratch. See Feldera documentation for details on the bootstrapping process.

post/v0/pipelines/{pipeline_name}/update_runtime

Request

  • The document declares no server URL.
  • Auth: HTTP bearer

Path parameters

pipeline_namestring required

Unique pipeline name

Response

Pipeline successfully updated

descriptionstring

Human-readable description of the pipeline. Default is an empty string.

tagsstring[]

Free-form labels used to organize, group, and filter pipelines. Default is no tags (empty vector).

created_atstring date-time required
deployment_desired_status'Stopped' | 'Unavailable' | 'Standby' | 'Paused' | 'Running' | 'Suspended' required
deployment_desired_status_sincestring date-time required
deployment_idstring uuid nullable
deployment_initial'Unavailable' | 'Coordination' | 'Standby' | 'Paused' | 'Running' | 'Suspended'
deployment_resources_desired_status'Stopped' | 'Provisioned' required
deployment_resources_desired_status_sincestring date-time required
deployment_resources_status'Stopped' | 'Provisioning' | 'Provisioned' | 'Stopping' required

Pipeline resources status.

/start (early start or provision check failed)
┌───┐
│   ▼
Stopped ◄────────── Stopping
/start │                   ▲
│                   │ /stop?force=true
│                   │ OR: timeout (from Provisioning)
▼                   │ OR: fatal runtime or resource error
⌛Provisioning ────────────│ OR: runtime status is Suspended
│                   │
│                   │
▼                   │
Provisioned ─────────────┘

Desired and actual status

We use the desired state model to manage the lifecycle of a pipeline. In this model, the pipeline has two status attributes associated with it: the desired status, which represents what the user would like the pipeline to do, and the current status, which represents the actual (last observed) status of the pipeline. The pipeline runner service continuously monitors the desired status field to decide where to steer the pipeline towards.

There are two desired statuses:

  • Provisioned (set by invoking /start)
  • Stopped (set by invoking /stop?force=true)

The user can monitor the current status of the pipeline via the GET /v0/pipelines/{name} endpoint. In a typical scenario, the user first sets the desired status, e.g., by invoking the /start endpoint, and then polls the GET /v0/pipelines/{name} endpoint to monitor the actual status of the pipeline until its deployment_resources_status attribute changes to Provisioned indicating that the pipeline has been successfully provisioned, or Stopped with deployment_error being set.

deployment_resources_status_detailsunknown
deployment_resources_status_sincestring date-time required
deployment_runtime_desired_status'Unavailable' | 'Coordination' | 'Standby' | 'Paused' | 'Running' | 'Suspended'
deployment_runtime_desired_status_sincestring date-time nullable
deployment_runtime_status'Unavailable' | 'Coordination' | 'Standby' | 'Initializing' | 'AwaitingApproval' | 'Bootstrapping' | 'Replaying' | 'Paused' | 'Running' | 'Suspended' | 'ConcurrentBootstrapping' | 'Synchronizing'

Runtime status of the pipeline.

Of the statuses, only Unavailable is determined by the runner. All other statuses are determined by the pipeline and taken over by the runner.

deployment_runtime_status_sincestring date-time nullable
deployment_status'Stopped' | 'Provisioning' | 'Unavailable' | 'Coordination' | 'Standby' | 'AwaitingApproval' | 'Initializing' | 'Bootstrapping' | 'ConcurrentBootstrapping' | 'Synchronizing' | 'Replaying' | 'Paused' | 'Running' | 'Suspended' | 'Stopping' required
deployment_status_sincestring date-time required
idstring uuid required

Pipeline identifier.

namestring required
platform_versionstring required
program_codestring required
program_status'Pending' | 'CompilingSql' | 'SqlCompiled' | 'CompilingRust' | 'Success' | 'SqlError' | 'RustError' | 'SystemError' required

Program compilation status.

program_status_sincestring date-time required
program_versioninteger required

Version number.

refresh_versioninteger required

Version number.

storage_status'Cleared' | 'InUse' | 'Clearing' required

Storage status.

The storage status can only transition when the resources status is Stopped.

Cleared ───┐
▲       │
/clear │       │
│       │
Clearing   │
▲       │
│       │
InUse ◄───┘
udf_ruststring required
udf_tomlstring required
versioninteger required

Version number.

Example response

{
  "deployment_error": {
    "error_code": "CodeSpecifyingErrorType",
    "message": "Explanation of the error that occurred."
  },
  "runtime_config": {
    "clock_timezone_offset": "+05:30"
  }
}

Changes

  • 522743adde02Revision changes
    • breaking

      added subschema #4 to the ///////////// response property oneOf list for the response status

    • breaking

      added subschema #4 to the ///////////// response property oneOf list for the response status

    • info

      added the optional property /////// to the response with the status

  • 0d17b7914d74Revision changes
    • info

      added the optional property /////// to the response with the status