Testing Spatial Data Pipelines

Spatial pipelines fail in a way ordinary data pipelines do not: the output is plausible. A wrong reprojection produces coordinates in the right hemisphere. A dropped overlay produces a parcel with fewer requirements rather than an error. A mishandled ring produces a polygon whose area is negative but whose bounding box looks fine. None of these raise, none of them fail a row count, and none of them will be caught by a test suite that asserts the job exited zero and wrote some rows. That is why testing gets its own topic inside the Municipal Zoning Data Architecture & Compliance Frameworks area: the assertions that matter here are spatial, and almost none of them are the assertions a general Python test suite teaches you to write.

Prerequisites and operational context

Three structural properties decide whether a spatial pipeline is testable at all, and retrofitting them is much harder than starting with them.

Time, network, and randomness must enter through parameters. A transform function that calls datetime.now() inside its body cannot be tested for effective-date behaviour, and a fetch function that hard-codes a URL cannot be tested at all. This is the same separation argued for in scheduling & orchestration: the scheduler owns the clock, and everything below it receives instants as arguments. In practice this single property is worth more to a spatial test suite than any testing library.

Geometry has to be comparable deterministically. Two polygons that are topologically identical can have different WKB bytes, so assert result == expected is either too strict or, once someone “fixes” it with a tolerance, too loose. The suite needs a canonical form — normalised ring order, fixed vertex start, rounded to a declared precision — and it needs to be the same canonical form the pipeline uses for change detection & geometry diffing, or the tests are validating a different definition of equality than production uses.

Fixtures must be small enough to read. A test whose fixture is a 40 MB county extract cannot be reasoned about when it fails: nobody can look at the input and say what the answer should be. Useful spatial fixtures are three to eight parcels with coordinates a human can check by hand, constructed to contain exactly the pathology under test.

Architecture: four test layers, each catching what the others cannot

Spatial correctness is not one property, so it is not one kind of test. The layers below are cumulative — each catches a class the others structurally cannot see.

Four test layers and the defect class each one owns Four stacked test layers with the defect class each one catches and the class it cannot see. Unit tests on pure functions catch parsing and arithmetic errors but cannot see whether a transformation preserves topology. Property-based tests over generated geometry catch invariant violations such as area changing under normalisation or a round trip failing, but cannot see whether the pipeline agrees with a known correct answer. Golden-file tests against a small hand-checked fixture catch regressions against a reviewed expected output, but only for the cases the fixture contains. Contract tests against a recorded portal response catch upstream schema drift, which none of the other three can see because they never touch the source. The diagram shows that no single layer is sufficient and that the cheapest layer catches the least interesting class of defect. Each layer is blind to the class below it in the right column 1 · unit tests on pure functions address normalisation, code parsing, backoff arithmetic, watermark decisions fast, hundreds of them, run on every save blind to: topology and projection 2 · property tests over generated geometry normalisation preserves area; round trips are identity; canonical form is stable finds the pathological polygon you would not have thought to write blind to: being confidently wrong 3 · golden-file tests on a hand-checked fixture six parcels, one known correct output, reviewed by a human once catches regressions precisely, and only for what the fixture contains blind to: upstream schema drift 4 · contract tests against a recorded portal response the real bytes a county returned, replayed offline, asserted field by field the only layer that sees a renamed column before production does blind to: nothing upstream of itself The layers are ordered by cost and inversely by consequence: layer 1 is cheapest and catches the least dangerous defects, layer 4 costs a stored fixture per source and catches the ones that reach production.
The ordering matters as much as the layers. Unit tests are the cheapest and catch the defects least likely to reach production; the contract test against a recorded portal response is the only layer that sees a renamed column before your users do.

The practical consequence is a suite whose shape is unusual: relatively few unit tests, a handful of property tests that do a great deal of work, one golden fixture per transformation, and one recorded response per source. Teams that build only layer 1 end up with a green suite and the failures described in the opening paragraph.

Production implementation

The two most valuable pieces are a canonical comparison helper and a property test over it. Everything else in a spatial suite tends to be built from these.

import math

import pytest
from shapely import wkb
from shapely.geometry import Polygon, mapping
from shapely.ops import orient

PRECISION = 3          # millimetre grid in metre-based CRSs; must match production
AREA_TOL_M2 = 1e-6


def canonical(geom):
    """Deterministic form for comparison: fixed ring orientation, rounded
    coordinates, and a fixed starting vertex. This must be the SAME function the
    change detector uses, or tests validate a different notion of equality than
    production does."""
    g = orient(geom, sign=1.0)                       # exterior counter-clockwise
    coords = [(round(x, PRECISION), round(y, PRECISION)) for x, y in g.exterior.coords[:-1]]
    start = min(range(len(coords)), key=lambda i: coords[i])
    ring = coords[start:] + coords[:start]
    holes = []
    for interior in g.interiors:
        hc = [(round(x, PRECISION), round(y, PRECISION)) for x, y in interior.coords[:-1]]
        hstart = min(range(len(hc)), key=lambda i: hc[i])
        holes.append(hc[hstart:] + hc[:hstart])
    return Polygon(ring, sorted(holes))


def assert_geom_equal(actual, expected, msg=""):
    """Compare canonically, and report WHY on failure — a byte diff of WKB is
    unreadable, so the failure message names the property that differs."""
    a, e = canonical(actual), canonical(expected)
    if a.equals(e):
        return
    reasons = []
    if abs(a.area - e.area) > AREA_TOL_M2:
        reasons.append(f"area {a.area:.4f} != {e.area:.4f}")
    if len(a.exterior.coords) != len(e.exterior.coords):
        reasons.append(f"vertex count {len(a.exterior.coords)} != {len(e.exterior.coords)}")
    if len(a.interiors) != len(e.interiors):
        reasons.append(f"hole count {len(a.interiors)} != {len(e.interiors)}")
    if not reasons:
        reasons.append(f"same area and vertex count but different coordinates; "
                       f"symmetric difference area = {a.symmetric_difference(e).area:.6f}")
    raise AssertionError(f"{msg or 'geometry mismatch'}: " + "; ".join(reasons))

The failure message is the part worth copying. A spatial assertion that fails with False is not True costs an hour; one that fails with hole count 1 != 2 costs a minute, and the difference compounds across a suite.

Property tests then exercise invariants rather than examples. The invariants below hold for every valid parcel polygon, which means a generator can look for counterexamples in places nobody would think to write a fixture for.

from hypothesis import given, settings
from hypothesis import strategies as st


@st.composite
def parcel_polygons(draw):
    """Simple closed quadrilaterals with realistic municipal coordinates, plus
    the degenerate shapes real county data actually contains."""
    x0 = draw(st.floats(min_value=500_000, max_value=600_000, allow_nan=False))
    y0 = draw(st.floats(min_value=4_000_000, max_value=4_100_000, allow_nan=False))
    w = draw(st.floats(min_value=0.5, max_value=400.0))
    h = draw(st.floats(min_value=0.5, max_value=400.0))
    skew = draw(st.floats(min_value=-50.0, max_value=50.0))
    return Polygon([(x0, y0), (x0 + w, y0 + skew), (x0 + w, y0 + h), (x0, y0 + h)])


@given(parcel_polygons())
@settings(max_examples=400, deadline=None)
def test_canonical_is_idempotent(poly):
    once = canonical(poly)
    assert canonical(once).equals_exact(once, 0), "canonical form is not stable"


@given(parcel_polygons())
@settings(max_examples=400, deadline=None)
def test_canonical_preserves_area(poly):
    assert math.isclose(canonical(poly).area, poly.area, rel_tol=1e-9, abs_tol=1e-6)


@given(parcel_polygons())
@settings(max_examples=200, deadline=None)
def test_reproject_round_trip(poly):
    """A reprojection out and back must land within the tolerance the pipeline
    claims. This is the test that catches a missing transform grid, because the
    approximation it silently substitutes does NOT round-trip cleanly."""
    projected = reproject(poly, "EPSG:26913", "EPSG:4326")
    back = reproject(projected, "EPSG:4326", "EPSG:26913")
    assert back.hausdorff_distance(poly) < 0.01, "round trip exceeded 1 cm"

test_canonical_is_idempotent looks trivial and is the test most likely to fail on a real implementation, because a normalisation that picks its starting vertex by anything other than a total order is unstable on polygons with repeated coordinates — which county data supplies in quantity.

Edge cases and gotchas

Tests that pass because the projection database is present. A CRS test on a developer machine with a full PROJ installation passes; the same test in a slim container fails, or worse, silently uses an approximation. Pin the PROJ data version, assert its presence in a session fixture, and include one test that deliberately asserts the failure path when a grid is missing — the behaviour described in handling EPSG lookup failures in pyproj.

Floating-point tolerance chosen by trial and error. A tolerance loosened until the suite goes green is a tolerance that no longer tests anything. Tolerances should be derived from the domain: a millimetre grid because survey data is not more precise than that, one centimetre for a round trip because that is the error budget the pipeline claims, two square metres for change detection because that is the noise floor. Each tolerance should have a comment naming its justification.

Fixtures that drift from production. A golden file generated by the current code asserts only that the code has not changed, not that it is right. The initial expected output has to be reviewed by a human against the ordinance or the source document once, and that review should be recorded in the fixture directory — a README naming who checked it and against what.

Property tests that generate impossible geometry. A generator producing self-intersecting bow-ties will fail every invariant, and the failure is the generator’s fault rather than the pipeline’s. Constrain generators to the shapes the pipeline is contracted to accept, and test pathological input explicitly as separate examples with expected rejections.

Speed collapse. Spatial operations in property tests are slow enough that a careless suite takes twenty minutes and stops being run. Keep the geometry small, cap example counts, and put the expensive whole-county work in a separate suite that runs on a schedule rather than on every commit.

Integration points

The test suite touches three other areas directly.

It shares the canonical form with change detection & geometry diffing. One implementation, imported by both — if the test suite has its own copy, the two definitions will diverge and the suite will certify behaviour production does not have.

It consumes recorded responses produced by the ingestion layer. The cheapest way to get layer-4 fixtures is for the fetch path to archive raw payloads anyway, which data lineage & provenance tracking already requires: the archive doubles as the contract-test corpus, so the marginal cost is a fixture loader rather than a new capture mechanism.

It gates deployment alongside the runtime gates. The distinction worth keeping clear is that schema validation & data quality checks validate data at run time while the suite validates code at build time. They overlap in what they assert and must not be collapsed: a runtime gate cannot tell you your reprojection is wrong, because the wrong coordinates are perfectly valid.

Compliance and audit artifacts

Testing produces two artifacts an audit can use, which is unusual for a test suite and worth exploiting.

The reviewed fixture is a compliance artifact in its own right: a small parcel set whose expected zoning outcome a named person checked against a named ordinance section on a known date. When a regulator asks how you know your setback computation is correct, a reviewed fixture is a better answer than a coverage percentage.

The contract-test corpus — recorded portal responses with their dates and digests — proves what each source was publishing when. That makes it the evidence behind a “the county changed their schema” claim, which is otherwise an assertion your team makes about a third party with nothing to back it.

Neither artifact needs to be created specially. Both fall out of testing the pipeline the way it has to be tested anyway, which is the strongest argument for doing it in this shape.

What to assert about a reprojection

Reprojection deserves its own assertions because it is the transformation most likely to be wrong and least likely to look wrong. Four checks cover almost all of it.

Four reprojection assertions against four ways a transform goes wrong A four-row by four-column matrix of reprojection failure modes against test assertions. Rows are a missing transform grid, a systematically shifted datum, feet mislabelled as metres, and a swapped axis order. Columns are the round-trip assertion, a known control point, an area-preservation check, and an explicit assertion that an undeclared CRS is refused. The round trip catches the missing grid but not the shifted datum, because a consistently wrong transform still returns to its starting point. Only the control point catches that. Four reprojection assertions against four ways a transform goes wrong round trip within tolerance known control point area preserved in equal-area undeclared CRS is refused Missing transform grid catches it catches it passes passes Systematically shifted datum passes catches it passes passes Feet mislabelled as metres sometimes catches it catches it passes Axis order swapped sometimes catches it catches it passes detected sometimes not detected
The round trip is the cheapest assertion and catches the most, but it is blind to a consistently wrong transform — reversing a systematic shift returns you to where you started. That blind spot is why one surveyed control point per jurisdiction earns its place in the suite.

The round trip is the cheapest and catches the most: transform out and back, and assert the result lands within the tolerance the pipeline claims. A missing transform grid fails this test, because the approximation PROJ silently substitutes is not its own inverse. This single test is worth more than any amount of eyeballing coordinates.

A known control point catches what the round trip cannot. A round trip through a consistently wrong transform still returns to its starting point, so it cannot detect a systematically shifted datum. A surveyed point whose coordinates you know in both systems will, and one such point per jurisdiction is enough.

An area-preservation check catches unit and axis-order mistakes. Reproject a polygon of known area into an equal-area projection and assert the area is preserved within a fraction of a per cent; feet-labelled-as-metres fails immediately, and so does a swapped axis order, which is otherwise almost impossible to spot in a coordinate pair that happens to look plausible.

Finally, an explicit failure test: assert that a batch with no declared CRS is rejected rather than transformed. Most CRS bugs in production are not wrong transforms — they are absent ones that were allowed through, so the test suite has to cover the refusal, not only the success.

Keeping the suite fast enough to be run

A spatial suite dies from slowness rather than from disagreement. Three habits keep it usable.

Where 46 spatial defects were caught, by layer Lollipop chart counting where 46 spatial defects were caught across four test layers and production. Unit tests caught 9, property tests 14, golden-file tests 11, contract tests against recorded portal responses 8, and 4 reached production. A dashed rule marks zero as the target for the production row. Property tests caught the largest share because they explore geometry nobody thought to write a fixture for. Where 46 spatial defects were caught, by layer 0 5 10 15 limit property tests 14 golden-file tests 11 unit tests 9 contract tests (recorded responses) 8 reached production 4 Defects caught at this layer Counts from one pipeline over eighteen months; the ranking is the point, not the totals.
Property tests caught the most, which is the argument for writing them even though they are the least familiar layer: they explore the pathological geometry nobody would think to put in a fixture. The four that reached production were all upstream schema changes arriving faster than the recorded-response corpus was refreshed.

Keep fixtures at three to eight parcels. A property test over a 40 000-parcel extract is not more rigorous than one over six carefully chosen shapes; it is slower and its failures are harder to read. Where whole-county behaviour genuinely needs exercising — index performance, memory ceilings — put it in a separate suite that runs nightly rather than on every commit.

Cap example counts deliberately, and set them from the shape of the input space rather than from a default. Four hundred examples over quadrilaterals with a skew parameter explores that space thoroughly; four hundred examples over a generator that produces one shape is four hundred identical tests.

And separate the layers in the runner, so a developer can run layers one and two in a couple of seconds and leave the recorded-response contract tests for the pre-push hook. A suite that takes twenty minutes is a suite that gets run once a day, which is the same as a suite that does not exist for the purpose of catching a mistake made two minutes ago.

FAQ

Why not just assert that two geometries are equal?

Because equality has several meanings for geometry and the default is usually the wrong one. Byte equality fails on cosmetic differences that do not change the shape, and topological equality passes on shapes whose coordinates have drifted within tolerance. Comparing canonical forms — fixed ring order, fixed starting vertex, declared precision — makes the assertion mean the thing you intended, and sharing that function with production keeps the test honest.

How large should a golden fixture be?

Small enough that a person can verify the expected output by hand, which in practice means three to eight parcels. The value of a golden file comes from the human review that established the expected output; a fixture too large to review is only asserting that the code has not changed, which a hash of the source would tell you more cheaply.

Do property tests replace example-based tests?

No — they cover a different class. Property tests establish that invariants hold across inputs you did not think of, which is where pathological county geometry lives. They cannot tell you that the pipeline agrees with a known correct answer, because they do not know what the answer is. Both layers are needed, and the property tests usually find the bugs while the golden files stop them coming back.

How do I test code that needs a projection database?

Pin the PROJ data version as a dependency and assert its presence in a session-scoped fixture, so a slim container fails loudly instead of silently substituting an approximation. Then test both paths: a transform that should succeed, and a transform requiring a grid you deliberately do not install, asserting that it is refused rather than approximated.