-
Continue reading →: Introduction and Getting Started with DBT on Databricks
As organizations continue to adopt modern data platforms, the need for reliable, scalable, and maintainable data transformation processes has become increasingly important. While Lakehouse platforms excel at storing and processing large volumes of data, transformation logic is often developed and maintained entirely by data engineers. In many organizations, even a…
-
Continue reading →: Demystifying Databricks Access Control
Access control is the most rudimentary form of maintaining security and data protection. It ensures that the right people have the right access to avoid any misuse of data. For example, in banking systems, customers can view only their own account and transaction details, while bank employees manage customer accounts,…
-
Continue reading →: Delta Sharing Introduction
When I want to read my Unity Catalog–governed data from a non-Databricks platform, the obvious solution is to set up ETL and load the data incrementally into the other platform. Databricks-provided Delta Sharing solves this problem by enabling data sharing without data copying or ETL setup for incremental loads. Delta…
-
Continue reading →: Architecting Static Outbound Connectivity for Azure Databricks
One of the most common challenges while working with Azure Databricks is dealing with its dynamic outbound IP addresses. If you’re trying to connect Databricks to external systems like Azure SQL or third-party APIs that require IP whitelisting, this quickly becomes a problem. In this article, I’ll walk through how…
-
Continue reading →: Databricks Identity Sync from Microsoft Entra ID
Identity management is essential for any application to ensure that the right people have the right level of access with the appropriate permissions. When using Azure as the cloud provider with Databricks, Microsoft provides built-in integrations that simplify identity and access management. In Databricks, identities such as users, groups, and…
-
Continue reading →: Secrets Management in Azure Databricks
Managing secrets is a core part of any application. Hardcoding secrets directly in notebooks or code is highly vulnerable. Therefore, systems provide secure ways to store secrets and use them when and where required, without exposing them directly in the code. Databricks provides a feature called Secret Scope, where we…
-
Continue reading →: Databricks SQL Introduction
If you are already using Databricks and thinking about moving to another platform just to get data warehouse capabilities, it might be worth reconsidering. Databricks SQL provides powerful data warehousing capabilities directly on top of your existing data lake. It is a collection of services designed to bring data warehouse…
-
Continue reading →: Databricks Serverless Compute
Databricks Serverless Compute is a fully managed compute option where Databricks automatically provisions, scales, and manages the infrastructure — you don’t create or manage clusters at all. Before setting up serverless compute, let’s understand where it fits within the Databricks architecture. Let’s look at the Databricks high-level architecture. The diagram…
-
Continue reading →: Processing ADLS delta-table using DuckDB
Modern data teams prioritize fast insights with minimal operational overhead. When your data already lives in Azure Data Lake Storage (ADLS) as Delta tables, spinning up Spark just to do light processing often feels like overkill. That’s where DuckDB shines. In this article, we’ll walk through processing Delta tables stored…
-
Continue reading →: DeltaLake change tracking with CDF & Row Tracking
As we know, Delta Lake tables are designed for the lakehouse architecture, combining the flexibility of a data lake with data-warehouse capabilities such as ACID transactions. Delta Lake also provides strong data-governance features, especially for tracking data changes. Two of them are Change Data Feed and Row Tracking, which we…
