Lakeflow Spark Declarative Pipelines is managed a framework for creating batch and streaming data pipelines in SQL and Python.

Why Lakeflow Spark Declarative Pipelines Matter

The core role of a data engineer is to implement business logic to ingest, transform, and serve data in a consumable way. Along with this, engineers must also manage dependencies, incremental processing, data quality rules, and SCD handling—responsibilities that are essential but repetitive.

Lakeflow Spark Declarative Pipelines take ownership of these concerns. Engineers simply declare what needs to be built, and the platform—powered by Databricks—decides how to execute it, including orchestration, state management, and scaling.

This matters because experienced engineers know scalability is critical. With fewer lines of code to maintain, teams focus only on core logic, reduce operational risk, lower data team costs, and rely on the platform to handle execution complexity.

Common Confusion between Delta Live Tables and Delta Tables

1. Lakeflow Spark Declarative Pipelines (SDP) is the new name for Delta Live Tables (DLT). And also referred as ETL in databricks world.
2. DLT / SDP is a managed data pipeline, not a table
3. Delta Tables are an open table format designed for the Lakehouse architecture.
4. SDP pipelines are powered by Delta Tables internally. But delta Tables can be used independently without SDP
5. SDP focuses on how data is built and managed while Delta Tables focus on how data is stored and accessed

Here it is what SDP covers

1. Managed workflow to load into the tables
2. Dependency management and orchestration
3. Incremental processing
4. Integrated data quality
5. Simplified SCD handling
6. Manage batch and streaming in the pipeline
7. Infra management – Cluster, libraries setup, auto scaling
8. Optimization
9. Operational efficiency – smaller codebase, fewer moving parts, reduced operational risk, reduced custom logic for retries, checkpoints and state management.
10. Idempotent, fault-tolerant, and recoverable by design
11. built-in monitoring, lineage, and pipeline health tracking

SDP important components

Pipelines –

1. A pipeline is meant for a Lakeflow spark declarative pipelines. In others words a pipeline is a definition of one SDP along with source code.
2. It can contain many flows, streaming tables, materialized views and sinks
3. Once the pipeline is defined with code, it analyses and decides what to do with it is own efficient plan.

Streaming table –

1. A streaming table is a form of Unity Catalog managed table that is also a streaming target for Lakeflow SDP.
2. A streaming table can have one or more streaming flows (Append, AUTO CDC) written into it.
3. Streaming tables are designed for append-only data sources and process inputs only once.

Materialized view –

1. A materialized view is also a form of Unity Catalog managed table and is a batch target. 
2. It is results of query & a flow to update it.
3. Unlike standard views it caches the results and refreshes so queries against it much faster than regular views.
4. Tracks changes in upstream data. On trigger, incrementally processes the changed data and applies the necessary transformations.
5. All required data is processed, even if it arrives late or out of order.
6. They are often incremental but not all. Databricks will try to choose the appropriate strategy that minimizes the cost of updating a materialized view.
7. Some changes to inputs will require a full computation of materialized view which can be expensive.
8. They are not designed for low-latency use cases.

Sinks –

1. A sink is a streaming target for a pipeline.
2. It can have one or more streaming flows (Append) written into it.

Important points

1. A table owned by a pipeline can be read by others, but it cannot be written to outside that pipeline. It applies to both streaming table and materialized view.
2. Adding or removing a table in the code, will automatically create and deletes the table, after the next run.
3. Deleting the pipeline will delete all the tables and views created by the respective SDP’s. Because it owns that.
4. Standard views can be created by @pipeline.view while materialized view and streaming table can be created by @pipeline.table

Getting started with spark declarative pipeline in databricks

Prerequisite –

1. Databricks workspace with premium plan & unity catalog enabled. Refer my other article Azure Databricks setup with unitycatalog
2. Databricks CLI is installed and authenticated on the base machine;
3. Setup the initial code base with databricks bundle, so that it can be deployed through databricks asset bundles and version control can be done through CI/CD. Refer the article to get familiar with bundle initialize and setting up initial code base – Deploying Lakeflow Jobs with Databricks Asset Bundles

Execution –

Resources spin up

Databricks CLI authentication

Source code for basic SDP

https://github.com/ArulrajGopal/kaninipro/tree/main/databricks_SDP_startup

Fully developed and deployed pipeline after first run.

After processing, pipeline graph.

Final output in the materialized view.

Reference for how tables/views are managed by pipeline.

Closing thoughts
Lakeflow Spark Declarative Pipelines shift data engineering from managing pipelines to declaring intent. By offloading orchestration, scalability, and reliability to the platform, teams can focus on what truly matters—building accurate, maintainable, and business-driven data products.

References

https://learn.microsoft.com/en-us/azure/databricks/ldp/

https://spark.apache.org/docs/latest/declarative-pipelines-programming-guide.html