A cleanly running Databricks job for three months just failed, and it’s time to dig into the results of each step to understand where it went wrong. Someone changed how duplicate orders get deduplicated — and nobody noticed, because nothing was watching. That’s the scenario a proper test suite exists…

“Why sudden drop in the record count, while the incoming stream and data size is unaffected?” is the most common question in the data engineering world. Now, we need to backtrack and understand the run details along with the processed data: was there any bulk data received with nulls in…

While data has become a valuable asset for every organization and plays a key role in the decision-making process, the value of analytical workloads has increased significantly as well. However, many organizations still have legacy data that has not yet been migrated and continues to reside in relational databases such…

While developing any data pipeline, it starts with extracting data from a source. Especially when preparing data for a BI dashboard or AI model, both need data from different sources. The source is less likely to be cloud storage and more likely to be enterprise SaaS applications like Salesforce, SAP,…

Whether the goal is to generate business insights, build AI-powered applications, or train machine learning models, organizing data into a structured format—rows and columns—is essential. Structured data offers several advantages: It is easier to analyse. It improves clarity and readability. It can be queried efficiently using SQL, the lingua franca…

Data governance is a paramount concern for every data engineer to ensure that data maintains high integrity. One of the key aspects of achieving this is granting access only to the right people. In Databricks, access control is implemented across multiple layers: the account level, the workspace level, and the…

Row-Level Security (RLS) is one of those database concepts that sounds intimidating—until someone explains it the right way. Think of it like a lunchbox rule at school. Everyone can open the fridge, but you can only take your lunchbox—not your friend’s, not your teacher’s. A database works the same way.…