Pandera vs Pydantic vs Great Expectations for parcel contracts

The three libraries are usually presented as alternatives, which is why teams pick one and then discover it cannot express half of what they need. They differ in their unit of work. Pydantic validates a single record and raises on the first bad one. Pandera validates a whole frame and reports every failing row. Great Expectations validates a dataset against a suite and produces a document about it. A parcel feed needs all three kinds of check — per-record types, cross-row uniqueness, and dataset-level distribution — so the useful question is which library owns which boundary. This guide places them, extending schema validation and data quality checks.

Diagnosis: sort the checks by their unit of work

Write down every check a parcel feed needs, then label each with the smallest amount of data that can decide it.

One record decides it. parcel_id is a non-empty string. zone_code is in the taxonomy. area_sqft is positive. The geometry parses. These are type-and-range checks and they belong at the parse boundary, where the record arrives, so a bad record can be quarantined without loading anything else.

One batch decides it. parcel_id is unique. No two parcels overlap by more than the noise floor. The set of zone codes present is a subset of the taxonomy. These need the whole frame in hand.

Nine parcel checks, sorted by the smallest data that decides them A nine-row by three-column matrix placing parcel-feed checks against the three units of work. Rows are identifier type, zone code in the taxonomy, positive area, geometry parses, identifier uniqueness, zone codes a subset of the taxonomy, no unexpected columns, row count within five percent of last week, and null rate not doubled. Columns are one record, one batch and the dataset over time. Each check is decidable at exactly one unit, and no single library covers all three. Nine parcel checks, sorted by the smallest data that decides them one record one batch the dataset over time Identifier is a string decidable — — Zone code in the taxonomy decidable — — Area is positive decidable — — Geometry parses decidable — — Identifier is unique — decidable — Zone codes are a subset — decidable — No unexpected columns decidable decidable — Row count within 5% of last week — — decidable Null rate has not doubled — — decidable Pydantic, at the parse boundary Pandera, before the upsert Great Expectations, on a schedule
Each check is decidable at exactly one unit, and the three units map to three libraries. Picking one library means the checks outside its unit either go unwritten or get expressed awkwardly — a uniqueness check threading a global set through a per-record validator.

The dataset over time decides it. The record count is within 5% of last week’s. The proportion of parcels with a null zone code has not doubled. The bounding box still matches the jurisdiction. These are drift checks and they need history to compare against.

A library that owns the wrong boundary produces awkward code: per-row uniqueness checks that need a global set threaded through, or a frame-level library invoked on single records.

Head-to-head

Pydantic Pandera Great Expectations
Unit of work one record one DataFrame a dataset + a suite
Failure report first error, or all field errors for that record every failing row, as a frame a validation result document
Cross-row checks no yes (unique, wide checks) yes
Historical comparison no no yes, via a store
Coercion yes, and central to its design yes, opt-in per column no
Runs on Python objects pandas, polars, pyspark pandas, SQL, Spark
Overhead per record microseconds (Rust core in v2) vectorised — near-zero per row high; not a per-batch tool
Natural home the parse boundary after load, before upsert a scheduled data-quality job

Step-by-step implementation

1. Pydantic at the parse boundary 

from typing import Annotated
from pydantic import BaseModel, Field, field_validator, ConfigDict

ZONE_CODES = frozenset({"R-1", "R-2", "R-3", "C-1", "C-2", "MU", "I-1", "AG"})


class ParcelRecord(BaseModel):
    """One record, straight off the feed. Rejections here go to quarantine with the
    raw payload attached, so a parser fix can replay them."""
    model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)

    parcel_id: Annotated[str, Field(min_length=1, max_length=32)]
    zone_code: str
    area_sqft: Annotated[float, Field(gt=0)]
    geometry_wkt: str

    @field_validator("parcel_id")
    @classmethod
    def preserve_leading_zeros(cls, v: str) -> str:
        # The identifier is a string, never an int. '0714221' coerced to a number and
        # back becomes '714221', which matches nothing.
        if not v.isascii() or not v.replace("-", "").isalnum():
            raise ValueError(f"unparseable identifier: {v!r}")
        return v

    @field_validator("zone_code")
    @classmethod
    def in_taxonomy(cls, v: str) -> str:
        code = v.upper()
        if code not in ZONE_CODES:
            raise ValueError(f"zone code not in taxonomy: {v!r}")
        return code

extra="forbid" is the schema-drift alarm. When a county adds an OVERLAY_CD column, a permissive model ignores it and the overlay silently never reaches the database; a forbidding model fails the batch and someone reads the field name.

2. Pandera after the load, before the upsert 

import pandera.pandas as pa
from pandera.typing import Series


class ParcelFrame(pa.DataFrameModel):
    """The whole batch. Reports every failing row rather than the first, which is what
    makes a 40,000-row feed diagnosable."""

    parcel_id: Series[str] = pa.Field(unique=True, str_matches=r"^[A-Za-z0-9-]{1,32}$")
    zone_code: Series[str] = pa.Field(isin=sorted(ZONE_CODES))
    area_sqft: Series[float] = pa.Field(gt=0, le=5.0e8)
    jurisdiction_id: Series[str] = pa.Field(nullable=False)

    class Config:
        strict = True          # unexpected columns fail, same reasoning as extra=forbid
        coerce = False         # coercion belongs at the parse boundary, not here

    @pa.dataframe_check
    def zone_mix_is_plausible(cls, df) -> bool:
        """A batch that is 98% one zone code is usually a parse failure that mapped
        everything to a default, not a genuinely uniform jurisdiction."""
        if len(df) < 500:
            return True
        return df["zone_code"].value_counts(normalize=True).iloc[0] < 0.95


def validate_batch(df):
    try:
        return ParcelFrame.validate(df, lazy=True)     # lazy: collect ALL failures
    except pa.errors.SchemaErrors as exc:
        # failure_cases is a frame of (schema_context, column, check, failure_case,
        # index) — join it back to the batch to quarantine exactly the bad rows.
        quarantine_rows(df, exc.failure_cases)
        raise

lazy=True is the flag that makes Pandera worth using. Without it, validation stops at the first failing check and a feed with three problems takes three runs to characterise.

Runs needed to characterise a feed with several problems Grouped bar chart of how many validation runs are needed to see every problem in a feed, against the number of distinct problems present. With lazy validation off, the count rises one for one — one run per problem, up to five runs for five problems. With lazy validation on, it is always one run regardless of how many problems there are. Runs needed to characterise a feed with several problems 0 2 4 6 1 1 1 problem 2 1 2 3 1 3 4 1 4 5 1 5 Validation runs to see every problem lazy validation off lazy validation on
Without lazy=True each run reveals one problem, so characterising a feed takes as many runs as it has faults — and each run is a full ingestion cycle. One flag turns a day of iteration into a single report.

3. Great Expectations for drift, on a schedule 

# Not per batch — per day, against the loaded table, comparing to history.
suite = context.add_or_update_expectation_suite("parcel_drift")

validator.expect_table_row_count_to_be_between(
    min_value=lambda: int(0.95 * last_week_count()),
    max_value=lambda: int(1.05 * last_week_count()),
)
validator.expect_column_proportion_of_unique_values_to_be_between(
    "parcel_id", min_value=1.0, max_value=1.0,
)
validator.expect_column_values_to_not_be_null("zone_code", mostly=0.98)
validator.expect_column_quantile_values_to_be_between(
    "area_sqft", quantile_ranges={"quantiles": [0.5], "value_ranges": [[4000, 12000]]},
)

The distinction that matters: none of these can fail a single batch, because none of them are about a batch. A 6% drop in row count is not an error in any record — it is a signal that a county changed something, and the response is investigation rather than rejection.

4. The layout that follows 

def ingest(raw_records, jurisdiction_id):
    """Three boundaries, three libraries, one direction of travel."""
    accepted, quarantined = [], []
    for raw in raw_records:                      # 1. Pydantic: per record
        try:
            accepted.append(ParcelRecord.model_validate(raw))
        except ValidationError as exc:
            quarantined.append((raw, exc.errors()))

    df = to_frame(accepted, jurisdiction_id)
    df = validate_batch(df)                      # 2. Pandera: per batch

    with conn.transaction():
        upsert_parcels(conn, df)
        insert_quarantine(conn, quarantined)
    # 3. Great Expectations runs later, on the table, comparing against history.

Most pipelines need Pydantic and Pandera. Great Expectations earns its considerable setup cost once there are enough jurisdictions that drift detection cannot be eyeballed — somewhere around a dozen sources, in practice.

Where each library sits in one ingestion cycle Timeline of one ingestion cycle across six stages, showing which library owns each. Fetching and decoding occupies the first stage with no validation. Pydantic validates per record across the parse stage. Pandera validates the assembled frame before the upsert. The upsert and quarantine insert commit together. Great Expectations runs afterwards on a separate schedule, comparing the loaded table against history. Where each library sits in one ingestion cycle 0 20 40 60 80 100 fetch and decode no validation yet parse per record — Pydantic rejections keep the raw payload assemble the frame validate frame — Pandera lazy: every failing row at once upsert + quarantine (one txn) accepted + quarantined = published drift suite — Great Expectations a separate schedule, vs history Progress through one ingestion cycle Only the green stage writes; the two validation stages decide what reaches it.
The two per-cycle libraries sit either side of frame assembly, and they commit together with the quarantine insert so that accepted plus quarantined always equals what the source published. Great Expectations sits outside the cycle entirely — it compares against history, which a single batch does not have.

Verification & testing

Validation code needs its own tests, because a validator that accepts everything passes every pipeline test silently.

def test_pydantic_rejects_an_unmapped_zone_code():
    with pytest.raises(ValidationError, match="not in taxonomy"):
        ParcelRecord.model_validate(GOOD | {"zone_code": "R-99"})


def test_pydantic_rejects_an_unexpected_field():
    """The schema-drift alarm. If this passes, a new county column is being dropped."""
    with pytest.raises(ValidationError):
        ParcelRecord.model_validate(GOOD | {"OVERLAY_CD": "HD-1"})


def test_leading_zeros_survive():
    assert ParcelRecord.model_validate(GOOD | {"parcel_id": "0714221"}).parcel_id \
        == "0714221"


def test_pandera_reports_every_failure_not_the_first():
    df = frame_with(bad_zone_rows=3, duplicate_id_rows=2, negative_area_rows=4)
    with pytest.raises(pa.errors.SchemaErrors) as exc:
        ParcelFrame.validate(df, lazy=True)
    checks = set(exc.value.failure_cases["check"])
    assert len(checks) >= 3          # all three problems, one run


def test_duplicate_ids_are_caught_by_pandera_not_pydantic():
    """Uniqueness cannot be decided by one record — this is the boundary in a test."""
    dup = [GOOD, GOOD]
    assert all(ParcelRecord.model_validate(r) for r in dup)      # both pass
    with pytest.raises(pa.errors.SchemaErrors):
        ParcelFrame.validate(to_frame(dup, "wa-king"), lazy=True)


def test_uniform_zone_mix_is_rejected():
    df = frame_with(rows=1000, all_zone="R-1")
    with pytest.raises(pa.errors.SchemaErrors):
        ParcelFrame.validate(df, lazy=True)


def test_taxonomy_is_shared_between_the_two_schemas():
    """Two copies of the code list drift. This asserts one source of truth."""
    assert set(ParcelFrame.to_schema().columns["zone_code"].checks[0].stats["allowed_values"]) \
        == ZONE_CODES

Failure recovery

A permissive model that dropped a column for months. Set extra="forbid", run the current feed, and read the errors — they name every field the county has added since. Recover the values by replaying quarantined payloads if you have them, or by re-fetching the source if the portal serves history.

Pandera failing an entire batch for twelve bad rows. That is the correct default for a contract violation, but not for a partial feed. Split the response: quarantine the rows named in failure_cases and upsert the rest, so one malformed downtown block does not hold a county’s worth of updates.

Great Expectations firing on a legitimate change. An annexation genuinely moves the row count by 8%. Record the expected change as a documented exception with an expiry, rather than widening the threshold permanently — a widened threshold stops detecting the thing it was added for.

Frequently asked questions

Can Pandera replace Pydantic entirely?

For batch work, largely — Pandera checks types, ranges and membership per column, and reports failures per row. What it does not do is validate a record before a frame exists. If the pipeline parses records one at a time and quarantines the bad ones, there is no frame to validate yet, and per-record validation has to live somewhere. Pipelines that read whole files into frames first can reasonably use Pandera alone.

Is Great Expectations worth the setup for one county?

No. For one source, the drift checks are three SQL queries in a scheduled task and the comparison baseline is a small table you write yourself. Great Expectations pays off when the same suite runs against many jurisdictions and somebody other than the pipeline author needs to read the results — the generated documentation is the actual product.

Where should coercion happen?

At the parse boundary, once, in Pydantic — and then nowhere else. Coercion later in the pipeline means two components disagree about what a field’s type is, and the one that loses is usually the identifier column, where a numeric coercion strips a leading zero. Pandera’s coerce is off in the config above for exactly that reason.

What about a JSON Schema contract instead?

Useful as the interoperable declaration of what a feed should look like, especially when a county or a downstream consumer needs to read it without running Python. It expresses per-record structure well and cross-row constraints not at all, so it complements rather than replaces the frame-level layer. Pydantic can emit JSON Schema from the model, which keeps one definition authoritative.