-
Continue reading →: Databricks Column Level Security
Data governance is a paramount concern for every data engineer to ensure that data maintains high integrity. One of the key aspects of achieving this is granting access only to the right people. In Databricks, access control is implemented across multiple layers: the account level, the workspace level, and the…
-
Continue reading →: Databricks Row Level Security
Row-Level Security (RLS) is one of those database concepts that sounds intimidating—until someone explains it the right way. Think of it like a lunchbox rule at school. Everyone can open the fridge, but you can only take your lunchbox—not your friend’s, not your teacher’s. A database works the same way.…
-
Continue reading →: Understanding Databricks UDFs
In any data platform, functions play a major role in keeping code modular, reusable, and clean. The Databricks ecosystem also provides functions, it is important to design functions correctly without compromising Spark’s capabilities. Like other systems, Databricks offers a rich set of built-in functions. However, this blog focuses on User-Defined…
-
Continue reading →: Introduction and Getting Started with DBT on Databricks
As organizations continue to adopt modern data platforms, the need for reliable, scalable, and maintainable data transformation processes has become increasingly important. While Lakehouse platforms excel at storing and processing large volumes of data, transformation logic is often developed and maintained entirely by data engineers. In many organizations, even a…
-
Continue reading →: Demystifying Databricks Access Control
Access control is the most rudimentary form of maintaining security and data protection. It ensures that the right people have the right access to avoid any misuse of data. For example, in banking systems, customers can view only their own account and transaction details, while bank employees manage customer accounts,…
-
Continue reading →: Delta Sharing Introduction
When I want to read my Unity Catalog–governed data from a non-Databricks platform, the obvious solution is to set up ETL and load the data incrementally into the other platform. Databricks-provided Delta Sharing solves this problem by enabling data sharing without data copying or ETL setup for incremental loads. Delta…
-
Continue reading →: Architecting Static Outbound Connectivity for Azure Databricks
One of the most common challenges while working with Azure Databricks is dealing with its dynamic outbound IP addresses. If you’re trying to connect Databricks to external systems like Azure SQL or third-party APIs that require IP whitelisting, this quickly becomes a problem. In this article, I’ll walk through how…
-
Continue reading →: Databricks Identity Sync from Microsoft Entra ID
Identity management is essential for any application to ensure that the right people have the right level of access with the appropriate permissions. When using Azure as the cloud provider with Databricks, Microsoft provides built-in integrations that simplify identity and access management. In Databricks, identities such as users, groups, and…
-
Continue reading →: Secrets Management in Azure Databricks
Managing secrets is a core part of any application. Hardcoding secrets directly in notebooks or code is highly vulnerable. Therefore, systems provide secure ways to store secrets and use them when and where required, without exposing them directly in the code. Databricks provides a feature called Secret Scope, where we…
-
Continue reading →: Databricks SQL Introduction
If you are already using Databricks and thinking about moving to another platform just to get data warehouse capabilities, it might be worth reconsidering. Databricks SQL provides powerful data warehousing capabilities directly on top of your existing data lake. It is a collection of services designed to bring data warehouse…
