Pipelines

Run the generated load

Finalizing and generating do not move any data. Kenseme designs the destination and writes the load code. You deploy that code to your own database or lakehouse and run it there, on your schedule and with your own credentials.

In the app: Data Model › Schemas › destination schema

[screenshot]
A destination schema version page with Load Script selected in the script-mode dropdown next to Generate Script.

Why generation and execution are separate

  • Credentials. Kenseme does not need write access to your production data.
  • Change control. Your normal review and deployment process runs the SQL, not Kenseme.
  • Safety. You choose the environment, the time, and the backup plan.

Get the code out of Kenseme

Pick whichever fits how you deploy.

One table at a time. On the destination schema’s Pipeline tab, open a table’s preview and click Copy to Clipboard.

Browse the objects. Open the destination schema’s Procedures tab (SQL Server, Fabric Warehouse) or Scripts tab (Databricks, Fabric Lakehouse), tick Show Generated, and open an object.

As a deployable bundle (recommended). Scripts are generated from a version of the destination schema:

  1. On the destination schema’s Info tab, click Create Version after you finalize and regenerate.
  2. Open that version.
  3. In the script-mode dropdown, pick a mode and click Generate Script:
    • Create Objects — tables, indexes, constraints, and your own views and procedures. No load code.
    • Load Script — only the generated load objects: helper views, per-table loads, and the master. It assumes the tables already exist.
    • Full Script — both in one file.
  4. Download the result.

See Generate a create script for the dialog’s options, including environment selection.

On Databricks and Fabric Lakehouse the result is a notebook (.ipynb) instead of a SQL file. See PySpark notebooks.

Deploy and run

SQL Server and Fabric Warehouse

  1. Run the Create Objects script against the destination database.
  2. Run the Load Script to create the helper views and procedures.
  3. Execute the master, [etl].[usp_Load_LoadAll], or a single table’s [etl].[usp_Load_<Table>].

Every procedure parameter is optional. Useful ones include @Debug, @BatchId for batch-based loads, and @WatermarkOverride for watermark-based loads.

Databricks and Fabric Lakehouse

  1. Import and run the Create Objects notebook.
  2. Import and run the Load notebook. It creates and refreshes the helper materialized views, defines each load function and the master. The last cell, Run (optional — uncomment to execute), holds the call to the master commented out, so a notebook-wide Run All never executes the load by accident. Uncomment it, or call the master from your own job, to run the load.

What the load expects

  • The source tables are reachable from where the load runs, as the environment qualifiers you chose point them.
  • The run-log table zz_<destination schema>_Jobs exists in the etl schema. The create script includes it.
  • Audit columns such as LoadDate and RowHash are filled in by the load. You do not map them.

When a run fails

  • SQL Server and Fabric Warehouse procedures run each table’s load in a transaction and roll it back on error.
  • Spark has no multi-statement transactions. Each step is a single Delta operation, and the notebook stops at the first failed cell.
  • Check your platform’s own logs and the zz_<destination schema>_Jobs table. For generated SQL that will not run, see Troubleshooting generated SQL or Troubleshooting the PySpark notebook.

Tip: Keep the generated scripts in source control and deploy them through your usual pipeline. You get review and history for free.