Pipelines

Keep a Pipeline in sync with its source

Source schemas keep changing after you build a Pipeline. Sync from Source compares the Pipeline with the current sources, adds what is new, and flags what has gone. Mappings whose source has gone are called stale, and a Pipeline with stale mappings has drift.

In the app: Data Model › Pipelines › your Pipeline › Sync from Source

[screenshot]
The mapping editor with the drift banner, the Resolve stale mappings button, and a stale table row highlighted.

Before you start

  • Your organization must not be read-only. Sync, resolve, and re-finalize are disabled in a read-only organization.

Sync from source

  1. Open the Pipeline’s mapping editor.
  2. Click Sync from Source.
  3. If nothing changed, you see Everything up to date — no changes found.

  4. Otherwise the Sync Summary dialog lists what happened:

    Line Meaning
    N tables added, N views added New source tables and views, now included with the Truncate & Reload strategy.
    N columns added New columns on tables you already map.
    N columns rebound A column was dropped and re-created with the same name. The mapping now points at the new column and keeps your destination name and settings.
    N tables flagged stale, N columns flagged stale The source no longer has them.

    When anything is stale, a Drift detected section lists each item.

Sync keeps your destination names, keys, load strategies, and calculated-column choices. A table that is dropped and re-created with the same name is re-linked the same way columns are, instead of being flagged stale.

If the Pipeline was finalized and sync changes anything, it goes back to Mapping In Progress.

Recognize drift

  • On the Pipelines page, the card shows a Drift detected badge. Hover for the last sync time.
  • In the mapping editor, a banner reads, for example, 3 stale mappings — Sync ran <date>.
  • Stale table and column rows are tinted and carry a Stale tag. Hover it for the reason.
  • Validation reports each stale mapping as an error, so you cannot finalize until you deal with it.

Resolve stale mappings

Stale mappings point at sources that no longer exist, so the fix is to remove them.

  1. In the drift banner, click Resolve stale mappings.
  2. The Resolve Stale Mappings confirmation says how many mappings will be removed. This cannot be undone.
  3. Click Resolve.
  4. You see, for example, Resolved 1 table and 2 columns.

Tables that lost a column have their load code marked Stale so you remember to regenerate it.

Tip: When a source table is renamed, sync adds it under the new name as a new table and flags the old name stale. Set up the new mapping’s destination name, keys, and strategy to match the old one before you resolve the stale mapping.

Re-finalize

Re-finalize appears in the toolbar when a finalized Pipeline has drift or out-of-date load code. It runs the catch-up steps in order:

  1. Syncs from the source.
  2. Validates. If there are errors, the Validation Results dialog opens and it stops there.
  3. Opens the Finalize Pipeline dialog so you can confirm.

Re-finalize does not regenerate the load code. Afterwards, open the destination schema’s Pipeline tab and click Regenerate All.