Pipelines

Generate the load procedures

Each table in a Pipeline gets its own load code: a stored procedure on SQL Server and Fabric Warehouse, or a PySpark function on Databricks and Fabric Lakehouse. You generate it from the destination schema, preview it, and regenerate it whenever the mappings change.

In the app: Data Model › Schemas › destination schema › Pipeline tab

[screenshot]
The destination schema's Pipeline tab with Regenerate All, Generate Master Proc, and the Proc status column showing Current and Stale rows.

Before you start

  • Finalize the Pipeline so the destination tables exist. See Finalize a Pipeline.
  • Give each table a load strategy and complete its configuration. Tables with strategy None are skipped. See Choose a load strategy.
  • The destination must be SQL Server, Fabric Warehouse, Databricks, or Fabric Lakehouse. For PostgreSQL, MySQL, Snowflake, and Redshift, generation stops with Procedure generation is not supported for dialect ….
  • Your organization must not be read-only. Preview still works in a read-only organization.

Generate all tables

  1. Open the destination schema and select the Pipeline tab.
  2. Click Regenerate All.
  3. A progress dialog, Generating procedures (or functions), counts through the tables.
  4. When it finishes it reports, for example, 12 generated, 1 skipped (no strategy). Click Close.

Each row’s status column then shows Current.

Generate or preview one table

Each row ends with three buttons:

  • Gear — opens the strategy configuration.
  • Eye — opens Preview — <table>, showing the code Kenseme would generate. Nothing is saved.
  • Refresh — generates this table only. You see, for example, Procedure generated: [etl].[usp_Load_Customer] (184 lines).

In the preview dialog:

  • The Environment section lists the source and destination schemas. When they have environments configured, pick which one each should point at, or use Match environment to pick the same environment name for both. The code updates as you change it. Environments are set on each schema’s Settings tab; see Schema settings.
  • Copy to Clipboard copies the code.

Keep the code current

The status column (Proc, or Function on Databricks and Fabric Lakehouse) shows:

Status Meaning
Not Generated No load code yet.
Stale The mappings, strategy, or settings changed since the code was generated.
Current The code matches the current mappings.

A N stale badge in the header counts out-of-date tables. Kenseme does not regenerate for you. Click Regenerate All, or the row’s refresh button.

Build the master procedure

The master calls every table’s load in order, so you run one thing instead of many.

  1. Generate the table loads first. The master button stays disabled until at least one exists.
  2. Click Generate Master Proc (on Databricks and Fabric Lakehouse, Generate Master Function).
  3. You see Master procedure generated: … orchestrates N procedure(s) (N lines).

Regenerate the master after you add or remove tables.

What gets generated

Destination Per-table load Master
SQL Server, Fabric Warehouse Stored procedure [etl].[usp_Load_<Table>] [etl].[usp_Load_LoadAll]
Databricks, Fabric Lakehouse Script usp_Load_<Table> holding the PySpark function load_usp_Load_<Table> Python function usp_LoadAll

The etl schema and usp_Load_ prefix are the defaults.

Some tables also get helper views:

  • vw_hash_<source table> for hash-based strategies that compute the row hash.
  • vw_promo_<source table> for calculated columns produced through a view.

On Databricks and Fabric Lakehouse these views are created as materialized views and refreshed in order before the loads run. See PySpark notebooks.

Where the generated code lives

Everything is saved into the destination schema as ordinary objects:

  • Stored procedures on the Procedures tab.
  • PySpark functions on the Scripts tab.
  • Helper views on the Views tab.

Generated objects are hidden by default. Tick Show Generated on the tab to see them. If every object on a tab is generated, the tab says so.