Schema Validation & Data Quality Checks for Municipal GIS

Raw municipal feeds almost never arrive in production shape. A county portal renames ZONING_CD to zoning_classification between releases, a planning department ships a shapefile with self-intersecting parcel rings, an open-data export emits null effective dates, and a vendor quietly switches the coordinate reference system without touching the file name. Each of these is a silent failure: the record loads, the numbers look plausible, and the corruption only surfaces three stages downstream as a spatial join that returns nothing, a Floor Area Ratio that is off by orders of magnitude, or a compliance flag that fires on the wrong parcel. Within Municipal Zoning Data Architecture & Compliance Frameworks, schema validation is the deterministic gate that stops this class of error at the door — the contract that guarantees only structurally sound, topologically valid, semantically correct records reach the staging database and the PropTech systems that read from it.

The discipline here is to treat validation as a first-class pipeline stage with its own contract, its own failure routing, and its own audit output — not as a scatter of defensive if checks bolted onto the loader. A record either clears the gate and is committed, or it is quarantined with structured error metadata so a data steward can triage it without halting ingestion. This page covers how that gate is built: the attribute contract, the geometry and topology checks, the cross-portal normalization that has to run before validation can even compare fields, and the audit artifacts the whole thing must emit.

Prerequisites and operational context jump to heading

Validation is an early stage, but its guarantees are only meaningful when a few things are already in place upstream:

• A canonical record shape. Validation compares incoming fields against an expected model, so features should already be coerced into your municipal data structures before the contract runs. Validating raw vendor payloads directly couples your schema to every portal’s quirks.
• A settled projection. Geometry checks that measure area or test jurisdictional bounds are meaningless if the projection is unknown. The CRS alignment strategies gate must have run first so every geometry is in a known, metric-correct working CRS before topology and area thresholds are asserted.
• A field-name normalization layer. Because municipalities do not share field names or code vocabularies, an attribute normalization pass and the canonical zoning taxonomy mapping must translate incoming columns and district codes to the canonical vocabulary before the attribute contract executes — otherwise every cross-portal field looks like a violation.
• A quarantine path. A gate with no place to send rejects is just an exception that kills the run. A dead-letter table or quarantine directory must exist so failed records are preserved with their error metadata while compliant records keep flowing.
• Pinned validation libraries. Pydantic v2, Shapely, and GeoPandas behave differently across major versions; pin them and record the versions in the run manifest so a validation result is reproducible months later during an audit.

If these are missing, fix them first. Validation is a gate, and a gate that runs before normalization — or before the projection is known — rejects good data and passes bad data.

Architecture: a three-tier deterministic gate jump to heading

The core design decision is to make validation a sequence of gates a record must clear, each cheaper to run than the one after it, so the pipeline fails fast and never spends geometry-repair cycles on a record that was malformed at the byte level. Three tiers run in order, and a failure at any tier short-circuits to quarantine.

1. Syntax check. Validate JSON/CSV structure, encoding, and required headers. This catches truncated downloads, BOM/encoding corruption, and feeds whose column set changed entirely between releases — the cheapest failures to detect and the most catastrophic if missed.
2. Attribute contract. Enforce a strict data model: required fields, type casting, regex constraints on identifiers and codes, enumerated zoning classifications checked against the ordinance, and business rules such as effective dates falling inside a plausible planning horizon. This is where schema drift surfaces the moment a portal changes its model.
3. Spatial integrity. Run geometry validity, CRS corroboration, topology, ring orientation, and minimum-area checks. This is the most expensive tier, which is exactly why it runs last — only on records that already have a sound structure and a valid attribute payload.

Records that clear all three tiers are committed to staging with a validation_timestamp. Records that fail any tier are routed to quarantine with the original payload preserved alongside the failing tier, the error class, and a human-readable message, so a steward can replay or repair them. Transient causes — a network blip mid-parse — are retried through the error handling & retry logic layer rather than quarantined as permanent failures.

Validation tier reference jump to heading

Each tier has a distinct failure signature and a distinct triage owner. Treating them separately keeps the quarantine queue legible instead of a single undifferentiated pile of “bad records.”

Defining rigid attribute contracts jump to heading

The foundation of reliable ingestion is an explicit schema contract rather than implicit type coercion or dynamic dictionary parsing. A rigid model maps directly to the canonical municipal data structure and rejects anything that does not conform. Modern Python validation frameworks like Pydantic provide strict type enforcement, regex constraints, and custom validators that isolate malformed payloads before they reach spatial processing.

A production attribute validator must mandate the fields downstream compliance logic depends on — parcel identifiers, zoning classifications, effective dates, jurisdictional codes — and constrain their values: reject nulls in critical fields, enforce ISO 8601 dates, and validate enumerated zoning codes against the ordinance. When a municipality updates its portal or amends its ordinance, the contract immediately flags the drift instead of silently absorbing it.

from pydantic import BaseModel, Field, field_validator, ConfigDict
from datetime import date
from typing import Optional
import logging

logger = logging.getLogger(__name__)


class ZoningFeatureSchema(BaseModel):
    # strict=True disables silent type coercion: "12" will not become 12
    model_config = ConfigDict(strict=True)

    parcel_id: str = Field(..., min_length=10, max_length=20,
                           description="Unique municipal parcel identifier")
    zoning_code: str = Field(..., pattern=r"^[A-Z]{2,4}-\d{2,4}$",
                             description="Canonical zoning classification")
    effective_date: date
    jurisdiction: str = Field(..., min_length=2)
    geometry_type: str = Field(..., pattern=r"^(Polygon|MultiPolygon)$")
    land_use_category: Optional[str] = None

    @field_validator("effective_date")
    @classmethod
    def validate_planning_horizon(cls, v: date) -> date:
        # Dates before modern zoning records or in the future indicate
        # a parsing error or a swapped day/month field, not real data.
        if v < date(1980, 1, 1) or v > date.today():
            raise ValueError("Effective date outside municipal planning horizon")
        return v

Enforcing geospatial and topological integrity jump to heading

Attribute validation alone is insufficient for spatial pipelines. Municipal shapefiles and GeoJSON exports frequently contain self-intersecting polygons, sliver geometries, and unclosed rings that break spatial indexing and overlay operations. Geometry checks run immediately after the attribute contract, leveraging Shapely and GeoPandas to enforce OGC Simple Features compliance.

Topology validation verifies that every geometry is valid, non-empty, and correctly typed. For zoning layers, additional checks matter: minimum-area thresholds to filter digitization artifacts, coordinate-bounds validation to confirm features fall inside the jurisdiction’s known extent (the same corroboration the CRS gate performs, repeated here as a defensive assertion), and ring-orientation enforcement — counter-clockwise exteriors per the GeoJSON right-hand rule. These gates prevent the spatial-join failures and wrong area math that would otherwise reach developers and planners as confident, incorrect numbers.

from shapely.geometry import shape
from shapely.validation import make_valid
from typing import Optional, Tuple

# Minimum parcel area in the working CRS units (square meters).
MIN_PARCEL_AREA_M2 = 10.0


def validate_geometry(geo_dict: dict) -> Tuple[bool, Optional[str]]:
    try:
        geom = shape(geo_dict)
    except Exception as e:
        # Unparseable geometry — malformed coordinates or wrong structure.
        return False, f"Geometry parse failure: {e}"

    if geom.is_empty:
        return False, "Empty geometry"

    if not geom.is_valid:
        # Attempt an OGC-valid repair (closes rings, splits self-intersections).
        geom = make_valid(geom)
        if geom.is_empty or geom.geom_type not in ("Polygon", "MultiPolygon"):
            return False, "Geometry collapsed or changed type after repair"

    if geom.area < MIN_PARCEL_AREA_M2:
        return False, "Geometry below minimum area threshold"

    return True, None

Cross-portal consistency and taxonomy alignment jump to heading

Municipalities rarely share identical data models. One city uses ZONING_CD, another zoning_classification, and the underlying classification codes diverge even when the field names happen to match — R-1 means single-family detached in one jurisdiction and something subtly different in the next. Reconciling these discrepancies requires a translation layer that maps incoming fields and codes to the canonical schema before validation runs, so the contract compares apples to apples. This is where validation is tightly coupled to taxonomy work: lookup tables and rule-based transformers normalize disparate municipal codes into the unified vocabulary the attribute contract enforces.

When scaling across jurisdictions, the gate must load jurisdiction-specific schema variants dynamically. A configuration-driven validation registry lets teams swap schema definitions without touching core pipeline code — each city registers its field map, its date format, and its enum set, and the gate selects the right variant at runtime. For the full multi-municipality treatment, including version-pinned schema variants and drift detection across portals, see validating zoning schema consistency across city portals.

Production implementation jump to heading

The orchestrator below integrates the three tiers, structured logging, and quarantine routing into a single deterministic stream processor. It preserves the raw payload of every rejected record for forensic replay, tags each rejection with the tier that failed, and emits a per-record validation_timestamp on everything it passes. It is written to run over a generator so a county feed of millions of features is processed without materializing the whole set in memory.

import json
import logging
from pathlib import Path
from typing import Generator, Any
from datetime import datetime, timezone
from pydantic import ValidationError

# Structured logging for compliance audit trails.
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s | %(levelname)s | %(message)s",
)
logger = logging.getLogger("zoning_validator")


class ZoningValidationPipeline:
    def __init__(self, schema_version: str, quarantine_path: Path = Path("quarantine")):
        self.schema_version = schema_version
        self.quarantine_path = quarantine_path
        self.quarantine_path.mkdir(exist_ok=True)
        self.counts = {"passed": 0, "attribute": 0, "geometry": 0}

    def process_stream(
        self, records: Generator[dict[str, Any], None, None]
    ) -> Generator[dict[str, Any], None, None]:
        for idx, raw_record in enumerate(records):
            # Tier 2: attribute contract (tier 1 syntax handled by the loader).
            try:
                validated = ZoningFeatureSchema.model_validate(raw_record)
            except ValidationError as e:
                self.counts["attribute"] += 1
                self._quarantine(raw_record, idx, "attribute_violation", str(e))
                continue

            # Tier 3: spatial integrity.
            geom_valid, geom_err = validate_geometry(raw_record.get("geometry", {}))
            if not geom_valid:
                self.counts["geometry"] += 1
                self._quarantine(raw_record, idx, "geometry_violation", geom_err)
                continue

            # Enrich and yield a committed record.
            clean = validated.model_dump(mode="json")
            clean["validation_timestamp"] = datetime.now(timezone.utc).isoformat()
            clean["schema_version"] = self.schema_version
            self.counts["passed"] += 1
            yield clean

    def _quarantine(self, record: dict, idx: int, error_type: str, message: str) -> None:
        payload = {
            "original_record": record,
            "error_type": error_type,
            "message": message,
            "record_index": idx,
            "schema_version": self.schema_version,
            "timestamp": datetime.now(timezone.utc).isoformat(),
        }
        path = self.quarantine_path / f"error_{idx}_{error_type}.json"
        path.write_text(json.dumps(payload, indent=2))
        logger.warning("Record %d quarantined: %s | %s", idx, error_type, message)

    def report(self) -> dict:
        total = sum(self.counts.values())
        rate = self.counts["passed"] / total if total else 0.0
        summary = {**self.counts, "total": total, "pass_rate": round(rate, 4)}
        logger.info("Validation run complete: %s", summary)
        return summary

This pattern keeps validation deterministic, preserves raw payloads for forensic analysis, and maintains a strict separation between compliant and non-compliant streams. The report() summary is itself an audit artifact: a sudden drop in pass rate is the earliest signal that a portal changed its schema.

Edge cases and gotchas jump to heading

Production municipal data finds the corners no toy dataset exercises. The ones worth hardening against:

• make_valid changes geometry type. Repairing a self-intersecting polygon can yield a GeometryCollection mixing polygons and stray lines. Always re-check the type after repair and reject anything that is no longer a Polygon/MultiPolygon, as the code above does — silently keeping the collection corrupts every later overlay.
• Strict mode rejects numeric strings. With Pydantic strict=True, a parcel id arriving as integer 12345 from a CSV-to-JSON step fails type validation. Normalize types in the loader, not by loosening the contract — relaxing strictness to fix one feed reopens the door to silent coercion everywhere.
• Deprecated codes that still parse. A retired zoning code like R-1 may match the regex perfectly yet no longer exist in the current ordinance. Regex alone is not enough; the enum/lookup check against the live taxonomy is what catches it.
• Day/month swaps inside the horizon. 03/04/2024 passes the planning-horizon validator whether it means March or April. Pin the source date format per jurisdiction in the schema variant rather than relying on a permissive parser.
• Area thresholds in the wrong units. A MIN_PARCEL_AREA_M2 of 10 is meaningless if the geometry is still in degrees because the CRS gate was skipped. Geometry validation must assert the working CRS is set before it trusts an area number.
• Antimeridian and high-latitude bounds. Coordinate-bounds checks that assume a simple lon/lat box misfire near the antimeridian or at extreme latitudes. Use the jurisdiction’s actual envelope, not a global heuristic.
• Quarantine that fills the disk. A portal that flips its entire schema can quarantine an entire feed in one run. Cap the quarantine writer or sample beyond a threshold, and let the falling pass rate trip an alert instead of silently exhausting storage.

Integration points jump to heading

Validation sits in the middle of the pipeline and both consumes and produces contracts. Upstream, the ingestion layer relies on async batch processing to pull and reproject large county feeds into the canonical staging shape this gate expects; validation is the quality checkpoint that engine’s output must clear before anything is committed. Downstream, only validated, timestamped records flow into GIS export sync workflows, so the shapefile and GeoJSON artifacts published to PropTech consumers carry the guarantee that every feature passed the contract. Records the gate quarantines hand off to fallback routing logic, which decides whether to substitute a secondary source or hold the parcel for manual review, and persistent schema drift surfaced here feeds the policy mapping in compliance framework integration.

Compliance and audit artifacts jump to heading

For PropTech underwriting and regulatory review, a validation run is only defensible if it leaves a reconstructable trail. Every run must emit:

• A per-record disposition. Each feature is either committed with a validation_timestamp and schema_version, or quarantined with its original payload, failing tier, error class, and message. There is no third, undocumented state.
• A run-level summary. Total records, pass/fail counts by tier, and the pass rate — the artifact that lets an auditor confirm a given day’s feed was checked and shows when a portal’s schema shifted.
• Schema and library provenance. The pinned Pydantic/Shapely/GeoPandas versions and the schema-variant identifier used, so a result can be reproduced exactly during an audit months later.
• Quarantine retention. Rejected payloads kept long enough to satisfy the retention window, so a disputed parcel can be replayed against the contract that rejected it.

Together these turn validation from a silent filter into an accountable control: anyone reviewing an underwriting decision can trace the parcel back to the exact contract version that admitted or rejected it.

Validation checklist jump to heading

• Normalize field names and zoning codes to the canonical taxonomy before the attribute contract runs.
• Confirm the working CRS is set before asserting any area threshold or coordinate-bounds check.
• Enforce required fields, types, regex, enums, and a plausible date horizon with a strict Pydantic model.
• Re-check geometry type after make_valid; reject anything no longer a Polygon/MultiPolygon.
• Route every failure to quarantine tagged with the failing tier and a human-readable message.
• Retry transient parse/network failures rather than quarantining them as permanent.
• Emit a run summary (counts, pass rate) and pin/record library and schema versions for audit.

FAQ jump to heading

Related jump to heading

• Validating zoning schema consistency across city portals — multi-municipality schema variants and drift detection in depth
• CRS alignment strategies — settles the projection geometry checks depend on
• Zoning taxonomy mapping — normalizes district codes before the attribute contract runs
• Municipal data structures — the canonical record shape this gate validates against
• Fallback routing logic — handles records this gate quarantines
• Error handling & retry logic — retries transient failures instead of quarantining them

Up: Municipal Zoning Data Architecture & Compliance Frameworks