-
Continue reading →: DeltaLake change tracking with CDF & Row Tracking
As we know, Delta Lake tables are designed for the lakehouse architecture, combining the flexibility of a data lake with data-warehouse capabilities such as ACID transactions. Delta Lake also provides strong data-governance features, especially for tracking data changes. Two of them are Change Data Feed and Row Tracking, which we…
-
Continue reading →: Introducing Lakeflow Spark Declarative Pipelines
Before defining Lakeflow Spark Declarative Pipelines, let’s first understand the declarative approach, Spark declarative pipelines, and finally Lakeflow Spark declarative pipelines. Procedural vs declarative Any task in computer science that describes how the task should be performed is considered a procedural programming approach, whereas defining what needs to be achieved—leaving…
-
Continue reading →: SQL Queries that make the code simple
SQL is the most widely used language across data processing applications. For a qualified data engineer, writing efficient queries is a vital skill—but equally important is the ability to write simple, clean, and readable SQL that is easy to maintain over time. During my exploration of various SQL problems, I…
-
Continue reading →: Databricks data quality with declarative pipeline
Databricks Spark Declarative Pipelines go beyond simplifying pipeline maintenance—they also address data quality, which is paramount for any data application. Using expectations, you can define data quality checks that are applied to every record flowing through the pipeline. These checks are typically standard conditions, similar to what you would write…
-
Continue reading →: Schema Drift Made Easy with Spark Declarative Pipelines
Spark Declarative Pipelines are designed to simplify the way data processing applications are built by letting engineers work declaratively—you focus on what needs to be produced, and the platform takes care of how it gets executed. This approach also extends naturally to handling schema evolution. Whether you need to add…
-
Continue reading →: Incremental load (SCD 1 & 2) with Spark declarative pipelines
Incremental load is an efficient approach for moving data into downstream systems by ensuring that only the changes between the previous run and the current run are processed. However, setting this up is not trivial. There are multiple proven strategies—such as batch-based processing using watermarks to track progress, or streaming…
-
Continue reading →: Getting started with Databricks SDP
Spark Declarative Pipelines are one of the flagship capabilities of Databricks, enabling data engineers to focus purely on business logic while abstracting away infrastructure concerns such as cluster provisioning and management etc. In this article, we will explore how to get started with Spark Declarative Pipelines using Databricks. Prerequisite –…
-
Continue reading →: Tracking Table and Column Lineage in Databricks Unity Catalog
Data governance is one of the most integral parts of any data project, and data lineage plays a key role in understanding and tracking the true source of data. What is data lineage? Data lineage provides end-to-end visibility of how data moves across systems—from its origin, through every transformation, to…
-
Continue reading →: Deploying Lakeflow Jobs with Databricks Asset Bundles
Databricks Lakeflow Jobs provide a powerful way to orchestrate notebooks and data processes directly inside Databricks without relying on external orchestration tools like Azure Data Factory, Airflow, or Dagster. A key requirement for modern data engineering is keeping job definitions as code and deploying them consistently across environments. This is…
