Pipelines

Choose a load strategy

A load strategy decides how each run of the generated load treats a destination table: wipe and refill, add only new rows, update changed rows, or keep history. It is the biggest single choice that shapes the generated code.

In the app: Data Model › Schemas › destination schema › Pipeline tab

You can also change the strategy from the Load Strategy column of the mapping editor, but configuration, analysis, and generation live on the destination schema’s Pipeline tab.

[screenshot]
The Load Strategies grid on the destination schema's Pipeline tab, with one row showing the configuration-required warning icon.

Before you start

  • Mark key columns in the mapping editor. Every strategy except Truncate & Reload needs at least one column marked Is PK on the table. Ticking SK does not count: validation looks for a mapped key column.
  • Your organization must not be read-only.

The strategies

Strategy Pick it when Needs
Truncate & Reload The table is small or cheap to rebuild: lookups, reference data. This is the default for new tables. Nothing
Incremental Load The source has a trustworthy last-modified date or time column. A watermark column
Incremental Batch The source has an ascending batch or sequence number. A batch column
Incremental Hash Rows change but there is no reliable timestamp. Changed rows are found by comparing a hash, and rows missing from the source are flagged deleted. Nothing required; optionally an existing hash column
SCD Type 2 A dimension where you need the full history of changes. A change detection method
None You do not want a load generated for this table. —

For behavior, audit columns, and per-dialect notes, see Pipeline load strategies.

Set a strategy

  1. Open the destination schema and select the Pipeline tab. The tab appears only on a schema that a Pipeline writes to.
  2. In the Strategy column, pick a strategy.
  3. For every strategy except Truncate & Reload, the Configure: <table> dialog opens. Fill it in and click Save.

A warning icon next to the strategy means required configuration is missing. Click it, or the gear button at the end of the row, to open the dialog.

Configure the strategy

Field Shown for What it does
Change Detection Method SCD Type 2 How changed rows are found: Hash (compare row hash) (default), Watermark (datetime column), or Batch (sequence/batch ID).
Hash Source Incremental Hash, or SCD Type 2 with Hash Computed (Generated View) (default) builds a helper view that hashes each row. Existing Source Column uses a hash column the source already has.
Source Hash Column Existing Source Column The source column holding the row hash.
Watermark Column (source datetime) Incremental Load, or SCD Type 2 with Watermark The source date or date-time column. Only date and date-time columns are listed.
Lookback Minutes Same as above How far before the last watermark to re-read, to catch late rows. 0–1440, default 5.
Source Batch Column Incremental Batch, or SCD Type 2 with Batch The numeric batch or sequence column. Only integer and decimal columns are listed.
Handle Deletes, Handle Reactivations SCD Type 2 Close out rows that disappear from the source, and reopen rows that come back. Both on by default.

Save stays disabled until the required fields are filled. On success you see Strategy configuration saved.

Other columns on the Pipeline tab

  • Type — label a table as Dimension, Fact, Reference, or Staging for your own classification.
  • Detection — the change detection method of an SCD Type 2 table.
  • SK — the same surrogate key switch as in the mapping editor.
  • Proc (Function on Databricks and Fabric Lakehouse) — whether the load code is Not Generated, Stale, or Current. See Generate the load procedures.

Let Kenseme suggest strategies

Analyze All inspects each table’s columns and relationships and recommends a strategy. It is rule-based and does not use AI.

  • It only fills in tables that have no strategy yet. Tables you already set are left alone.
  • You see Analysis complete. N strategies auto-applied. or Analysis complete. No new recommendations.
  • If relationship data is missing, a warning says the analysis used partial data.

Note: Analyze All can recommend strategies that are not in the dropdown, such as SCD Type 1 or the fact strategies. A table with one of these shows an empty Strategy cell. 🚩 TODO: confirm how these should display and whether users can pick them.

Common picks

  • Currency, country, or status codes → Truncate & Reload.
  • A large transaction table with a ModifiedDate → Incremental Load.
  • A feed that stamps every row with a load or batch ID → Incremental Batch.
  • A customer table with no reliable timestamp → Incremental Hash.
  • A customer dimension where you must see past addresses → SCD Type 2.

What changes when you change a strategy

  • The table’s load code is marked Stale until you regenerate it.
  • A finalized Pipeline goes back to Mapping In Progress. Finalize again to add the audit columns the new strategy needs.