Incremental vs full exports for large parcel datasets

A county assessor’s parcel fabric holds 1.8 million polygons, and every night your job has to publish it to a dozen downstream consumers — a title company’s warehouse, two municipal dashboards, an impact-analysis service. Push the whole 4 GB snapshot every night and you burn bandwidth re-sending 1.79 million unchanged rows to move the 3,000 that actually changed; consumers re-index the world at 2 a.m. Push only the deltas and you inherit a harder problem: how does a consumer that missed last night’s file know a parcel was deleted, and how do you prove the incremental stream still reconstructs the same fabric as a full snapshot? This page frames that trade-off as a decision. It is a companion to the broader GIS export sync workflows that publish the outputs of the wider automated zoning change and municipal GIS tracking pipeline. The answer is rarely “always one” — it is a matrix, a hybrid, and a reconciliation check that proves the two agree.

Symptoms that force the decision jump to heading

You are usually pushed to reconsider your export mode by one of these operational pains rather than choosing it on day one.

1. Nightly full exports blow the maintenance window. The snapshot grows with the fabric; re-transferring and re-loading millions of unchanged rows starts overrunning the window and the downstream re-index.
2. A consumer’s data is silently stale. With incremental exports, a consumer that dropped one change-set never learns of parcels deleted in the gap, so retired PINs linger and split parcels double-count acreage.
3. Two consumers disagree. One rebuilt from Tuesday’s full snapshot, another applied Monday-plus-Tuesday deltas, and their parcel counts differ by a few hundred. Without a reconciliation contract you cannot tell which is right.

The core tension is that a full snapshot is self-describing — it is the complete truth as of an instant, deletions included by omission — while an incremental stream is cheap but stateful, correct only if every consumer applies every change-set in order and you communicate deletes explicitly with tombstones.

Decision matrix jump to heading

Score the two modes against the dimensions that actually bite in production. No single row decides it; weight them by your consumers.

When to pick which, and the hybrid jump to heading

Pick full snapshots when consumers are heterogeneous and you cannot guarantee they apply every change-set in order, when downstream needs immutable point-in-time versions for audit or diffing, or when nightly churn is a large fraction of the fabric. The self-describing nature of a snapshot — deletes by omission, no watermark to track — is worth the bandwidth when correctness dominates.

Pick incremental when the fabric is large, daily churn is small, and consumers are few and controlled enough to guarantee ordered application. A change-set stream is the right feed for a latency-sensitive consumer like a change detection and geometry diffing service that only cares about what moved.

The hybrid is usually correct at county scale. Publish a full immutable snapshot on a slow cadence — weekly, or monthly — as an anchor, and incremental change-sets between anchors. Consumers rebuild cleanly from the latest snapshot and fast-forward with deltas; a consumer that falls too far behind simply reloads the anchor instead of replaying weeks of change-sets. The heavy delta computation and multi-consumer fan-out belong on the async batch processing layer so a slow consumer never stalls the export.

Implementing an incremental change-set export jump to heading

Diff the current parcel table against the last published version by a stable content hash per row, emit upserts for new or changed geometry, and — critically — emit tombstones for PINs that vanished. Without tombstones, deletions are invisible.

import hashlib
import geopandas as gpd

def row_hash(geom_wkb: bytes, zoning: str, area: float) -> str:
    """Stable content hash so unchanged rows produce no delta."""
    h = hashlib.sha256()
    h.update(geom_wkb)
    h.update(zoning.encode())
    h.update(f"{area:.4f}".encode())
    return h.hexdigest()

def build_change_set(current: gpd.GeoDataFrame, previous: gpd.GeoDataFrame) -> dict:
    """Emit upserts for changed rows and tombstones for deleted PINs."""
    cur = {r.parcel_id: row_hash(r.geometry.wkb, r.zoning_code, r.geometry.area)
           for r in current.itertuples()}
    prev = {r.parcel_id: row_hash(r.geometry.wkb, r.zoning_code, r.geometry.area)
            for r in previous.itertuples()}

    upserts = [pid for pid, h in cur.items() if prev.get(pid) != h]
    tombstones = [pid for pid in prev if pid not in cur]  # deleted since last export

    change = current[current["parcel_id"].isin(upserts)].copy()
    change["_op"] = "upsert"
    return {
        "watermark": current.attrs.get("version", "unknown"),
        "upserts": change,
        "tombstones": tombstones,   # consumer must delete these PINs
    }

On the consumer side the merge must be ordered and idempotent — apply upserts, then process tombstones, keyed on parcel_id:

def apply_change_set(target: dict, change: dict) -> dict:
    """Idempotent merge: upsert by key, then remove tombstoned PINs."""
    for row in change["upserts"].itertuples():
        target[row.parcel_id] = row          # insert or overwrite
    for pid in change["tombstones"]:
        target.pop(pid, None)                # delete; no error if already gone
    target["_watermark"] = change["watermark"]
    return target

Implementing a full-snapshot export jump to heading

A snapshot is simpler and self-describing: write the entire fabric to an immutable, versioned file. GeoParquet keeps CRS metadata and compresses far better than Shapefile, and its columnar layout lets consumers diff two snapshots cheaply.

from datetime import datetime, timezone
import geopandas as gpd

def write_full_snapshot(parcels: gpd.GeoDataFrame, out_dir: str) -> str:
    """Write an immutable, versioned GeoParquet snapshot of the whole fabric."""
    version = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
    path = f"{out_dir}/parcels_{version}.parquet"
    # Preserve CRS in file metadata; never publish a snapshot without it.
    if parcels.crs is None:
        raise ValueError("refusing to write snapshot with undefined CRS")
    parcels.to_parquet(path, index=False, compression="zstd")
    return path  # immutable: never overwrite an existing version

Verification via row and geometry reconciliation jump to heading

Whichever mode you run, prove that a consumer built from change-sets reconstructs the same fabric as the full snapshot. Reconcile on both row count and a geometry-aware hash so a shifted vertex is caught, not just a missing row.

def reconcile(snapshot: dict, rebuilt: dict, tol: int = 0) -> dict:
    """Compare a full snapshot against a delta-rebuilt copy."""
    snap_keys, rebuilt_keys = set(snapshot), set(rebuilt)
    missing = snap_keys - rebuilt_keys          # consumer never got these
    extra = rebuilt_keys - snap_keys            # missed tombstone -> ghost PIN
    drift = [k for k in snap_keys & rebuilt_keys
             if snapshot[k]["geom_hash"] != rebuilt[k]["geom_hash"]]
    ok = len(missing) + len(extra) + len(drift) <= tol
    return {"missing": len(missing), "extra": len(extra),
            "geometry_drift": len(drift), "reconciled": ok}

• Row parity. missing and extra must both be zero. A non-zero extra is the classic dropped-tombstone ghost: a PIN deleted at the source that never left the consumer.
• Geometry parity. geometry_drift catches a re-surveyed polygon whose row survived but whose vertices moved — the same geometry-level comparison the change detection and geometry diffing service performs downstream.
• Watermark ordering. Assert each consumer’s stored watermark advances monotonically; a stalled or rewound watermark means a change-set was skipped or replayed out of order.

Failure recovery jump to heading

• Missed change-set → reload the anchor. A consumer whose watermark lags the oldest retained change-set cannot fast-forward; it must reload the latest full snapshot and resume deltas. Retain enough change-sets to cover your worst realistic consumer outage.
• Tombstone gap → periodic full reconcile. Because a single dropped tombstone leaves an undetectable ghost, run a full-snapshot reconciliation on a fixed cadence even in an incremental-primary setup, and delete any extra PINs the reconcile finds.
• Immutable snapshots, never overwrite. Publishing a new version must never mutate an existing file, so consumers can pin an exact point-in-time and roll back. A corrupted export becomes a new version, not an overwrite of a good one.
• Idempotent replay. Key both the change-set apply and the snapshot load on parcel_id so replaying a file after a mid-load crash converges to the same state rather than duplicating rows.
• Alert on drift. Emit watermark, missing, extra, and geometry_drift as structured logs and alert when any exceeds tolerance — a rising extra count is your early warning that tombstones are being lost.

Frequently asked questions jump to heading

Related jump to heading

• Parent topic: GIS Export Sync Workflows
• Section overview: Automated Feed Ingestion & GIS Data Parsing
• Async Batch Processing — computing deltas and fanning snapshots out to many consumers
• Change Detection & Geometry Diffing — the downstream consumer of change-sets and geometry-level reconciliation
• Automating shapefile and GeoJSON exports from municipal portals — the export mechanics these modes build on