Pipelines

Pipelines overview

A Pipeline takes one or more source schemas, maps their tables and columns into a new destination schema, and generates the code that loads data from one to the other. Use it to build a cleansed Silver layer from Bronze, or a Gold model from Silver, without hand-writing every load.

In the app: Data Model › Pipelines

[screenshot]
The Schema Pipelines page with two pipeline cards and the New Schema Pipeline button.

What a Pipeline is made of

  • Source schemas. One or more existing schemas in your organization. Their tables and views are the input.
  • A destination schema. Kenseme creates it when you create the Pipeline. It is an ordinary schema: you can browse it, version it, and generate scripts from it.
  • Table mappings. One row per source table or view, with the destination table name, whether it is included, whether to add a surrogate key, and a load strategy.
  • Column mappings. One row per source column, with the destination column name, primary-key flag, and, for calculated columns, how to handle the value.
  • Load strategies. Per table: Truncate & Reload, Incremental Load, Incremental Batch, Incremental Hash, or SCD Type 2.
  • A status. Mapping In Progress while you design, Finalized once the destination schema has been built from the mappings.

The journey

Each step has its own page. Work through them in order the first time.

  1. Create a Pipeline — pick sources, name the destination, set naming conventions.
  2. Map source columns to target columns — rename, include or exclude, set keys, review AI suggestions.
  3. Override a mapping with an expression — handle calculated columns.
  4. Choose a load strategy — decide how each table is loaded and configure it.
  5. Validate a Pipeline — fix errors before you commit.
  6. Finalize a Pipeline — build the destination tables and columns.
  7. Generate the load procedures — produce the code that moves the data.
  8. Run the generated load — deploy it to your database or lakehouse.
  9. Keep a Pipeline in sync with its source and re-run or roll back as things change.

What you get

  • Tables and columns in the destination schema, including surrogate keys you asked for and the audit columns each load strategy needs.
  • A load procedure per table (SQL Server, Fabric Warehouse) or a PySpark load function per table (Databricks, Fabric Lakehouse), plus a master procedure or function that runs them all.
  • Helper views when a strategy or a calculated column needs one.

Note: Kenseme designs and generates. It never connects to your database to create tables or load data. You run the generated scripts yourself. See Run the generated load.

Supported targets

You can pick any of the eight database systems for the destination schema. Load generation works for four of them:

Destination What Kenseme generates
SQL Server, Fabric Warehouse T-SQL stored procedures
Databricks, Fabric Lakehouse PySpark load functions, delivered in a notebook
PostgreSQL, MySQL, Snowflake, Redshift Mapping, validation, and finalize work. Load generation is not supported yet.

When to use something else

  • One-off fixes or backfills. Write the SQL directly.
  • A dimensional model with facts and dimensions. Start with a Star Schema plan. It builds Pipelines for you and links back to them.