Data Quality Rule Builder

Great Expectations (GX) is an open-source Python library for data quality testing. You define 'expectations' — assertions about your data — and GX validates them against your actual data. It integrates with pandas, Spark, SQLAlchemy, and cloud data warehouses. It generates HTML Data Docs reports and can be embedded in Airflow/Prefect pipelines to gate deployments when data quality fails.

dbt (data build tool) has a built-in testing framework. Tests are defined in YAML files alongside your models. Generic tests (not_null, unique, accepted_values, relationships) are built-in. Custom singular tests are SQL queries that return rows when the test fails. dbt test runs all tests and fails the build if any fail, giving you a quality gate in your CI/CD pipeline.

Start with the basics: 1) not_null on primary keys and required fields. 2) unique on primary keys. 3) accepted_values on enum/status columns. 4) range checks (min_value/max_value) on numeric fields like price, age. 5) regex patterns on structured strings like email, phone, postal code. 6) referential integrity (foreign key relationships). These cover 80% of data quality issues in production pipelines.

Options: 1) Fail-fast: halt the pipeline and alert on any failure (safest). 2) Soft alert: continue but send Slack/PagerDuty notification (for non-critical pipelines). 3) Quarantine: route bad rows to a dead-letter table for manual review. 4) Auto-repair: fix known issues (trim whitespace, fill nulls with defaults) before validation. Most teams start with fail-fast for critical pipelines and soft alerts for analytics.

Schema validation checks the structure (column names, data types, nullability). Data quality checks validate the content (values are within expected ranges, formats match patterns, referential integrity holds). Both are needed: schema validation catches structural drift (upstream systems changing column types), data quality checks catch business logic violations (negative prices, future birthdates, orphaned foreign keys).