A cleanly running Databricks job for three months just failed, and it’s time to dig into the results of each step to understand where it went wrong. Someone changed how duplicate orders get deduplicated — and nobody noticed, because nothing was watching.

That’s the scenario a proper test suite exists to prevent.

Stable data systems come with premium test framework integrated.

I put together a small repo with databricks ETL job along with a job for testing the functionality of the job. Nothing exotic: pytest, a couple of fixtures, and chispa for comparing Spark DataFrames. But wired together the right way, it catches exactly the kind of regression that ruins an evening.

Source code – [link]

Let’s demo.

Workflow setup

The project is an Orders ETL pipeline built on PySpark. It reads raw orders and customers data, cleans it up, joins the two together, and produces two outputs: an enriched, per-order dataset and a revenue summary per customer.

There is a two separate workflow, one is for regular and another is testing flow which does unit and integration tests.

And key factor design factor is every function is a plain DataFrame → DataFrame transformation with no side effects, none of them need a running cluster, a notebook, or even Databricks itself to be tested. That single design decision is what makes the rest of this post possible.

Testing setup

The test suite lives in tests/ and leans on two libraries: pytest for the test runner, and chispa for asserting that two Spark DataFrames are equal.

The spark fixture is session-scoped, so the whole test run shares one SparkSession instead of paying Spark’s startup cost per test.

tests/conftest.py defines the fixture that every test shares. It also defines raw_orders_df — a small, deliberately messy sample: a duplicate order_id with two different dates, one row with a missing order_id, mixed-case product names, and a row with null quantity and price. It’s messy on purpose, because clean sample data doesn’t tell you anything about whether clean_orders actually cleans.

unit tests – Each transformation function in isolation, take the dedup test as an example — it feeds the messy fixture through clean_orders and deduplicate_orders, and then checks two things: that only one row per order_id survives, and that it’s the newer of the two duplicate rows that got kept, not an arbitrary one.

Integration tests – steps back from individual functions and exercises the full chain — build_enriched_orders end to end, and a transform_and_write run that writes Parquet to a temporary directory and reads it back to confirm the output actually landed correctly on disk. It’s the difference between “each gear turns correctly” and “the whole machine produces the right output,” and a suite that only has one of the two tends to miss the bugs that show up at the seams.

Things not covered in testing –

Draw an honest line around what isn’t unit-tested. Auto Loader ingestion needs a real cluster to mean anything, so it’s left to the Databricks test job rather than a shallow local mock.

Snapshot of databricks testing job result

Outro

A test suite doesn’t need to be clever to be worth building — it needs to be watching. A few fixtures, one deliberately messy dataset, and a clear line between what’s tested and what isn’t were enough to catch the exact regression that once slipped through silently. That’s the real payoff: not smarter debugging after the fact, but the failure showing up before it ever reaches production.