Expectations in Lakeflow: data quality as code, governed in Unity Catalog

In most pipelines I inherit, "data quality" is a script that runs after the load: a battery of SELECT COUNT(*) WHERE field IS NULL that someone opens on Monday and, by the time it finds a problem, the bad data has already been consumed by three dashboards and a model. It's reaction, not prevention.

Lakeflow Expectations (the Databricks Declarative Pipelines, evolution of the old DLT) flip that logic around. The quality rule now lives next to the transformation, as versioned code, and Unity Catalog governs, audits and reuses that rule across pipelines. In this article I show the concept, the code in practice and why it matters for anyone running data in production.

The concept: quality as a contract, not a check

An Expectation is a declarative constraint on a pipeline dataset — a materialized view, a streaming table or a temporary view. You describe, in boolean SQL, what a valid record means and tell the pipeline what to do when the condition isn't met.

The mindset shift is the word contract. Instead of "I'll check later whether it came in null", you declare "this table does not accept a null ID" and the engine takes care of the rest — including recording how many records violated the rule, so you can measure the source's health over time.

The code in practice

The entry point in 2026 is the PySpark pipelines module. You import it, decorate the function that produces the table and attach the expectation:

from pyspark import pipelines as dp

@dp.table()
@dp.expect_or_drop("valid_id", "id IS NOT NULL")
def customers():
    return spark.readStream.table("bronze.customers_raw")

Three elements here:

• @dp.table() declares that the function materializes a pipeline table.
• @dp.expect_or_drop(...) attaches the rule. The first argument is the name of the expectation (it appears in the metrics), the second is the SQL condition a valid record must satisfy.
• The function returns the query — Spark handles the execution plan.

The 6 decorators and when to use each

The module offers six decorators, organized by what happens to the violation and by number of rules.

Single rule:

• @dp.expect — logs and keeps the row (monitor).
• @dp.expect_or_drop — drops the bad row.
• @dp.expect_or_fail — stops the pipeline.

Multiple rules (dictionary):

• @dp.expect_all — logs and keeps.
• @dp.expect_all_or_drop — drops.
• @dp.expect_all_or_fail — stops.

The choice is a business decision, not a code one:

• expect — use it when you want observability without blocking. Ideal to start: you measure the violation rate of a new source before deciding to block.
• expect_or_drop — use it when the bad row can't reach consumption, but the pipeline can continue with the rest. It's the most common case in silver/gold layers.
• expect_or_fail — use it for business-critical invariants (e.g. a duplicated primary key, a negative monetary value where it's impossible). Failing early is cheaper than propagating.

Multiple rules at once

To validate several conditions, use the _all variants, passing a name -> condition dictionary:

rules = {
    "valid_id":        "id IS NOT NULL",
    "plausible_age":   "age BETWEEN 0 AND 120",
    "state_filled":    "state IS NOT NULL",
}

@dp.table()
@dp.expect_all_or_drop(rules)
def customers():
    return spark.readStream.table("bronze.customers_raw")

With expect_all_or_drop, a record is dropped if it fails any of the rules. Each rule is still measured individually in the metrics — you know which condition is knocking data out.

The 2026 leap: rules governed by Unity Catalog

Historically, expectations lived tied to the pipeline code. The game-changing news is being able to store and manage the quality rules inside Unity Catalog tables. In practice, that brings three direct benefits:

1. Versioning and auditing — the rule stops being a line lost in a notebook and becomes a governed object, with history. Audit and compliance can answer "which rule was in effect in March?".
2. Reuse across pipelines — the same definition of "valid customer" can be referenced by several pipelines, instead of copy-pasted (and diverging over time).
3. Centralized governance — together with the automatic propagation of MANAGE permissions to materialized views and streaming tables in Unity Catalog, quality enters the same governance perimeter as the rest of the data.

Add to that other recent Lakeflow evolutions that make the pattern more robust in production: Python unit tests right in the pipeline editor (validating the logic against mocked data via table redirection), type widening to evolve column types without a pipeline reset, and a queued execution mode that queues concurrent updates instead of failing on conflict.

Why this matters

When quality becomes declarative, governed code, three things change for the team:

• Reliable data without babysitting the load. The rule runs on every execution; the violation is logged automatically. No one needs to open the dashboard on Monday morning.
• An explicit risk decision. expect / drop / fail force the team to decide, for each rule, the cost of letting it through versus the cost of blocking it. That documents the risk appetite in the pipeline itself.
• Native observability. The quality metrics are part of the pipeline — you can track the violation rate per source over time and act before it becomes an incident.

How to start tomorrow

1. Pick one critical silver table and list 3 to 5 invariants you know should hold.
2. Start with @dp.expect (monitor only) for a few days to measure the real violation rate — it avoids dropping good data over a poorly calibrated rule.
3. Promote the stable rules to expect_or_drop; reserve expect_or_fail for the few invariants that justify stopping the load.
4. When the pattern matures, move the definitions into Unity Catalog and share them across pipelines.

Quality stops being a weekly reaction and becomes a contract the pipeline honors on every run. In practice, it's the kind of change that tends to pay dividends faster than almost any performance optimization — because it prevents the silent rework of reprocessing bad data that's already been consumed.