Why I chose this topic: In my last three years managing HIPAA and PCI-compliant pipelines, I’ve seen more production outages caused by "dirty" data than by actual infrastructure failure. We need to stop treating data quality as a post-hoc dashboard exercise and start treating it as a hard network-layer gate.
Eighty-two percent of data engineers I’ve interviewed believe that "failing the pipeline" is the only way to ensure data integrity. They are dead wrong. If your clinical trial ingestion pipeline crashes because a provider sent a malformed JSON object at 3:00 AM, you haven't protected the data—you’ve denied the business the 99% of valid data they actually needed to make decisions.
In highly regulated environments, the "fail-fast" mantra is a luxury you can't afford. You need a "quarantine-first" architecture. If you are still using .fail(), you are building fragile pipelines that demand manual intervention. I’m going to show you how to leverage Delta Live Tables (DLT) expectations to isolate bad data without stopping the world.
1. Stop using fail expectations in production
The default behavior for most DLT users is to use the fail constraint. It’s seductive because it feels safe. But in a production environment handling millions of records, one stray null in a patient_id column shouldn't brick your entire ingestion flow.
Use expect_drop or just expect with a downstream filter instead. If you use expect, DLT keeps the record but flags it. If you use expect_drop, it discards it. For compliance, discarding isn't enough—you need to know what you dropped.
# The "Fail" pattern is for local testing only.
# Don't put this in your production CI/CD.
@dlt.expect_or_fail("valid_transaction_id", "transaction_id IS NOT NULL")
def transactions():
return spark.readStream.table("raw.transactions")
Photo by Haberdoedas on Unsplash
2. The Quarantine Sink Pattern
Instead of dropping data into the void, route your violations to a secondary "quarantine" table. This is non-negotiable for audit trails in healthcare. You need to prove to an auditor exactly which records failed validation and why.
Use the _expectations metadata column that DLT automatically populates. You can filter for rows where the expectation failed and sink them into a delta table configured for long-term storage.
@dlt.table(name="clean_transactions")
@dlt.expect_or_drop("valid_amount", "amount > 0")
def clean_transactions():
return dlt.read("raw_transactions")
@dlt.table(name="quarantine_transactions")
def quarantine_transactions():
# Filtering for rows that failed the 'valid_amount' expectation
return dlt.read("raw_transactions").filter(
"amount <= 0 OR amount IS NULL"
)
3. Versioning your expectation logic
Data contracts change. Last year, a valid SSN might have been 9 digits; this year, the legacy system might be throwing junk characters into that field. If you hardcode your expectations, you are creating a maintenance nightmare.
Treat your DLT expectations like code. Move your expectation definitions into a YAML config file or a central Python dictionary. This allows you to update validation rules across 50+ pipelines without touching a single DLT notebook.
# config.yaml
rules:
patient_id_format: "length(patient_id) == 10"
valid_currency: "currency_code IN ('USD', 'EUR', 'GBP')"
# In your DLT pipeline
rules = load_yaml("config.yaml")
@dlt.expect("patient_id_check", rules['patient_id_format'])
def process():
...
4. Monitoring the "Drop Rate"
If your pipeline is silently dropping 5% of records because of an expect_drop constraint, your quality gate is actually a data leak. You need a circuit breaker.
Configure your DLT alerts in Databricks SQL to trigger when the expect_drop count exceeds a threshold. I keep a dashboard that tracks row_count vs dropped_count. If dropped_count > 1% of total, the pipeline should warn the on-call engineer immediately, even if the pipeline itself is technically "healthy."
5. Metadata-driven observability
Never rely on the DLT UI for long-term auditing. DLT writes its own execution logs to the event_log table. You should query this table to build a history of which expectations are failing most frequently.
I’ve seen engineers spend days debugging a "bad data" issue that could have been resolved in seconds by querying the expectations detail in the event_log.
SELECT
event_time,
details:expectations:name as expectation_name,
details:expectations:failed_records as failed_count
FROM delta.`/system/events/`
WHERE event_type = 'flow_progress'
Photo by Brecht Corbeel on Unsplash
6. The "Grace Period" Rule
When deploying a new validation rule, don't just flip the switch to expect_or_drop. You will likely discover edge cases you didn't anticipate.
Start with a "logging-only" expectation. Let it run for 24-48 hours. Use the quarantine sink to review what would have been dropped. Only after you’ve verified the data matches your expectations should you upgrade the rule to expect_or_drop. This is the difference between a stable platform and a midnight wake-up call.
Conclusion
The goal of a data engineer in a regulated space isn't just to move data from A to B; it's to be the arbiter of trust. If you rely on the pipeline crashing to tell you there's a problem, you’ve already failed. By implementing a quarantine sink, externalizing your rules, and monitoring your drop rates, you turn your pipeline into a self-documenting audit machine.
You aren't just shipping tables; you're shipping reliability. When the auditors come knocking, will you be able to pull up a table of every invalid record you've rejected, or will you be scrambling to explain why your pipeline was down for six hours?
Cover photo by Joe Murray on Unsplash.
