Overview

Great Expectations (GX) brings testing discipline to data pipelines. It answers: “Does this data match what I expect?” before it’s loaded into production tables, preventing bad data from silently corrupting analytics.

Core Concepts

Expectation: A statement about data (e.g., “column email should never be null”)
Expectation Suite: Collection of expectations for a dataset
Validator: Runs expectations against actual data
Checkpoint: Reusable validation pipeline step

Basic Usage

import great_expectations as gx
import pandas as pd

context = gx.get_context()

# Load data
df = pd.read_parquet("s3://data-lake/orders/2026-04-02.parquet")

# Create a batch of data to validate
batch = context.sources.pandas_default.read_dataframe(df)

# Define expectations
batch.expect_column_to_exist("order_id")
batch.expect_column_values_to_not_be_null("order_id")
batch.expect_column_values_to_be_unique("order_id")
batch.expect_column_values_to_not_be_null("customer_id")
batch.expect_column_values_to_be_between("amount", min_value=0, max_value=100000)
batch.expect_column_values_to_match_regex(
    "email",
    r"^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$"
)
batch.expect_column_values_to_be_in_set(
    "status",
    ["pending", "completed", "cancelled", "refunded"]
)

# Validate
results = batch.validate()
print(f"Success: {results.success}")
print(f"Passed: {results.statistics['successful_expectations']}")
print(f"Failed: {results.statistics['unsuccessful_expectations']}")

Saving and Reusing Expectation Suites

# Save an expectation suite
context = gx.get_context()

suite = context.add_expectation_suite("orders_suite")

validator = context.get_validator(
    datasource_name="orders_datasource",
    data_asset_name="daily_orders",
    expectation_suite_name="orders_suite"
)

validator.expect_column_values_to_not_be_null("order_id")
validator.expect_column_values_to_be_between("amount", 0, 100000)
validator.expect_table_row_count_to_be_between(min_value=100, max_value=1000000)

# Save suite to disk/cloud
validator.save_expectation_suite(discard_failed_expectations=False)

Checkpoint: Validation in Pipelines

# Define a reusable checkpoint
checkpoint = context.add_checkpoint(
    name="orders_daily_checkpoint",
    validations=[
        {
            "batch_request": orders_batch_request,
            "expectation_suite_name": "orders_suite",
        }
    ]
)

# Run in Airflow/Prefect/etc.
results = checkpoint.run()

if not results.success:
    raise ValueError(f"Data quality check failed: {results}")

Airflow Integration

from great_expectations_provider.operators.great_expectations import GreatExpectationsOperator

validate_orders = GreatExpectationsOperator(
    task_id="validate_orders",
    datasource_name="orders_datasource",
    data_asset_name="daily_orders",
    expectation_suite_name="orders_suite",
    fail_task_on_validation_failure=True,
)

When to Add Data Validation

Add Great Expectations at: ingestion from external sources, after major transformations, before loading to production tables. Skip it for intermediate scratch tables. The goal is to catch data problems at the source, not after they’ve propagated through your warehouse.

Best For

Data engineering teams building ETL/ELT pipelines that ingest from external APIs, third-party vendors, or upstream teams
dbt users who want to complement dbt tests with richer statistical expectations (row counts, distributions, value ranges)
Regulated industries (finance, healthcare) where data quality audits and documentation are compliance requirements
ML teams detecting training data drift — validate that production data matches the distribution your model was trained on
Data platform teams who want auto-generated data quality dashboards shared with business stakeholders

Great Expectations vs. Alternatives

	Great Expectations	dbt tests	Pandera	Soda
Expectation types	100+ built-in	Basic (not null, unique, ref)	Python schema validation	SQL + cloud
Auto data docs	✅	Partial	✗	✅
Pipeline integration	Airflow, Dagster, Prefect	dbt only	Python code	Any
Learning curve	High	Low	Medium	Medium
Best for	Complex validation + docs	dbt-native testing	Python dataframes	SQL-heavy stacks

Use Great Expectations when you need rich, reusable expectation suites with stakeholder-facing data quality reports. Use dbt tests for simpler SQL-layer validation within a dbt project.

Great Expectations — Data Quality Validation & Testing Framework

✅ Pros

⚠️ Cons