Building golden-file tests for geometry transformations

The suite has a golden file. It was produced by running the transformation and saving the output, so what it asserts is that the code still does what it did on the day somebody committed it — which is useful for catching regressions and says nothing about whether the transformation was ever right. Six months later the pipeline is reprojecting into the wrong state plane zone, the golden file agrees with it perfectly, and the suite is green. This guide builds golden files that carry real authority: a small reviewed fixture, an expected output established against a source outside the code, and a comparison that reports why it differs. It is the concrete practice behind layer three of testing spatial data pipelines.

Diagnosis: is your golden file evidence or a snapshot?

Three questions separate the two, and a file that fails any of them is a regression detector rather than a correctness test.

Where did the expected output come from? If the answer is “the code,” the file cannot detect a wrong transformation. If it is “a surveyed control point,” “the county’s own published area figure,” or “a hand calculation a named person checked,” it can.

Where an expected output came from, and what it can prove A four-row by three-column matrix of sources for a golden file's expected output. Rows are the code's own output, a hand calculation on round coordinates, the county's published parcel area, and a surveyed control point published in both coordinate systems. Columns are whether the file can detect a regression, whether it can detect a transformation that was always wrong, and whether it can be re-verified later. Only the code's own output fails the middle column, which is the column that matters for reprojection. Where an expected output came from, and what it can prove detects a regression detects an always-wrong transformation can be re-verified The code’s own output yes no only against itself Hand calculation on round coordinates yes yes yes County’s published parcel area yes yes yes Surveyed control point yes yes yes yes weakly no
Every row detects a regression; only three of the four can detect a transformation that was never right. That middle column is the whole difference between a golden file and a snapshot, and reprojection is precisely where the distinction bites — a wrong transform produces perfectly self-consistent output.

Could a person verify it today? A golden file with 40 000 features cannot be re-reviewed, so when it fails nobody can tell whether the new output or the old expectation is wrong. In practice the fixture stops being authority and starts being an obstacle — and the usual resolution is to regenerate it, which quietly destroys whatever authority it had.

Does the failure message say anything? assert result == expected on WKB gives a diff of hex. A failure that names area differs by 4.2 m² or hole count 1 != 2 is actionable in a minute.

def audit_fixtures(fixture_dir):
    """Fixtures without a provenance note are snapshots, not golden files."""
    import json, pathlib
    rows = []
    for path in pathlib.Path(fixture_dir).glob("**/*.json"):
        meta = path.with_suffix(".provenance.json")
        rows.append({
            "fixture": path.name,
            "features": len(json.loads(path.read_text()).get("features", [])),
            "has_provenance": meta.exists(),
            "reviewed_by": json.loads(meta.read_text()).get("reviewed_by") if meta.exists() else None,
        })
    return rows

Step-by-step implementation

1. Build the fixture from a real pathology, at reviewable size 

# fixtures/reproject_state_plane/input.py
"""Six parcels chosen to exercise one transformation each. Coordinates are in
EPSG:26913 (NAD83 / UTM 13N) and were taken from a published county extract, then
translated to a round origin so a person can check the arithmetic by hand."""

from shapely.geometry import Polygon

PARCELS = {
    # A plain rectangle: the control case. 40 x 30 m = 1 200 m².
    "simple":        Polygon([(500000, 4400000), (500040, 4400000),
                              (500040, 4400030), (500000, 4400030)]),
    # Clockwise exterior ring — the winding-order case.
    "clockwise":     Polygon([(500100, 4400000), (500100, 4400030),
                              (500140, 4400030), (500140, 4400000)]),
    # A hole: area must be 1 200 − 100 = 1 100 m².
    "with_hole":     Polygon([(500200, 4400000), (500240, 4400000),
                              (500240, 4400030), (500200, 4400030)],
                             [[(500210, 4400010), (500220, 4400010),
                               (500220, 4400020), (500210, 4400020)]]),
    # A sliver 4 cm wide — survives normalisation, must not be dropped.
    "sliver":        Polygon([(500300, 4400000), (500300.04, 4400000),
                              (500300.04, 4400030), (500300, 4400030)]),
    # Duplicate consecutive vertices, which county exports contain in quantity.
    "dup_vertices":  Polygon([(500400, 4400000), (500420, 4400000), (500420, 4400000),
                              (500440, 4400000), (500440, 4400030), (500400, 4400030)]),
    # A deep flag lot: the shape that breaks negative buffers.
    "flag_lot":      Polygon([(500500, 4400000), (500506, 4400000), (500506, 4400060),
                              (500540, 4400060), (500540, 4400090), (500500, 4400090)]),
}

Six shapes, each with a stated reason for existing, and coordinates rounded so area is checkable mentally. That is the property that makes the fixture re-reviewable in a year.

Minutes to resolve a golden-file failure, by fixture size Lollipop chart of the time taken to resolve a failing golden-file test against the number of features in the fixture. A six-feature reviewed fixture resolves in about 4 minutes. A fifty-feature fixture takes about 25. A five-hundred-feature fixture takes about 90 and is usually regenerated instead of investigated. A forty-thousand-feature fixture is always regenerated, which takes 2 minutes and destroys the review the file depended on. Minutes to resolve a golden-file failure, by fixture size 0 20 40 60 80 100 limit 6 features (reviewed) 4 min 50 features 25 min 500 features 90 min 40 000 features (regenerated) 2 min Minutes to resolve a failure The 40 000-feature row is fast because nobody investigates it — they regenerate.
The bottom row is the trap: a huge fixture is fast to deal with because everybody regenerates it, and regeneration silently converts the file from evidence into a snapshot. Six features is slower to resolve and is the only size where the answer is trustworthy.

2. Establish the expected output from something other than the code 

// fixtures/reproject_state_plane/expected.provenance.json
{
  "reviewed_by": "j.doe",
  "reviewed_on": "2026-08-11",
  "method": "Areas computed by hand from the fixture coordinates and cross-checked against the county's published parcel area for parcel 0714-22-1 (1 199.6 m², rounding to 1 200). Reprojected coordinates verified against a surveyed control point (NGS PID AB1234) whose EPSG:26913 and EPSG:4326 coordinates are both published.",
  "expected_areas_m2": { "simple": 1200.0, "clockwise": 1200.0, "with_hole": 1100.0,
                         "sliver": 1.2, "dup_vertices": 1200.0, "flag_lot": 1380.0 },
  "tolerance_m2": 0.01,
  "tolerance_rationale": "Survey data is not more precise than a centimetre, and the transformation is asserted to preserve area to 1e-4 relative."
}

The method field is what turns a fixture into evidence. It is also what a reviewer reads first when the test fails in eighteen months.

3. Compare canonically, and explain the difference 

from shapely import wkb

PRECISION = 3          # millimetre grid, matching production
AREA_TOL = 0.01        # m², from the provenance rationale


def diff_reason(actual, expected) -> str | None:
    """None when equal; otherwise the FIRST property that differs, in plain words."""
    a, e = canonical(actual), canonical(expected)     # shared with production
    if a.equals(e):
        return None
    if abs(a.area - e.area) > AREA_TOL:
        return f"area {a.area:.4f} m² != {e.area:.4f} m² (Δ {a.area - e.area:+.4f})"
    if len(a.interiors) != len(e.interiors):
        return f"hole count {len(a.interiors)} != {len(e.interiors)}"
    na, ne = len(a.exterior.coords), len(e.exterior.coords)
    if na != ne:
        return f"exterior vertex count {na} != {ne}"
    if a.exterior.is_ccw != e.exterior.is_ccw:
        return "exterior winding order differs"
    sd = a.symmetric_difference(e).area
    return (f"same area, holes and vertex count, but the shapes differ: "
            f"symmetric difference {sd:.6f} m² — check for a translation or a "
            f"start-vertex mismatch")


def assert_matches_golden(name, actual, golden):
    reason = diff_reason(actual, golden[name])
    assert reason is None, f"{name}: {reason}"
What each comparison says when a hole goes missing A four-row by two-column matrix of comparison strategies against the failure message each produces when a transformation drops an interior ring. Rows are byte equality on WKB, shapely equals, an area comparison, and the property-by-property diff. Columns are the message a developer sees and roughly how long it takes to act on it. Byte equality produces a hex diff and takes an hour; the property diff names the hole count and takes a minute. What each comparison says when a hole goes missing message a developer sees time to act Byte equality on WKB b'\x01\x03…' != b'\x01\x03…' ~1 hour shapely .equals() False is not True ~30 min Area comparison area 1200.0 != 1100.0 ~5 min Property-by-property diff hole count 1 != 2 ~1 min actionable partial opaque
The same bug, four messages. assert result == expected on WKB yields a hex diff and an hour of bisection; naming the property that differs — hole count 1 != 2 — turns it into a one-minute fix. The comparison's job is not only to fail but to say why.

4. Make regeneration deliberate and visible 

# conftest.py
def pytest_addoption(parser):
    parser.addoption("--regenerate-golden", action="store_true",
                     help="rewrite golden files from current output — REQUIRES a "
                          "provenance update in the same commit")


@pytest.fixture
def golden(request):
    path = FIXTURES / "reproject_state_plane" / "expected.wkb.json"
    if request.config.getoption("--regenerate-golden"):
        # Regeneration is allowed and must be conspicuous: it invalidates the review.
        warnings.warn("regenerating golden files — update expected.provenance.json "
                      "with who re-verified the output and against what",
                      stacklevel=2)
        return Regenerator(path)
    return json.loads(path.read_text())

A flag plus a warning is a small thing, and it is the difference between regeneration being a decision and being the path of least resistance when the suite goes red.

Verification & testing 

@pytest.mark.parametrize("name", list(PARCELS))
def test_reprojection_matches_golden(name, golden):
    out = reproject(PARCELS[name], "EPSG:26913", "EPSG:2232")
    assert_matches_golden(name, out, golden)


def test_areas_match_the_reviewed_values(provenance):
    """The assertion that does not depend on the code's own output at all."""
    for name, expected in provenance["expected_areas_m2"].items():
        area = PARCELS[name].area
        assert abs(area - expected) <= provenance["tolerance_m2"], \
            f"{name}: fixture area {area} disagrees with the reviewed value {expected}"


def test_diff_reason_is_informative():
    a = Polygon([(0, 0), (10, 0), (10, 10), (0, 10)])
    b = Polygon([(0, 0), (10, 0), (10, 11), (0, 11)])
    assert "area" in diff_reason(a, b)


def test_hole_loss_is_reported_as_such():
    solid = Polygon([(0, 0), (10, 0), (10, 10), (0, 10)])
    holed = Polygon([(0, 0), (10, 0), (10, 10), (0, 10)],
                    [[(4, 4), (6, 4), (6, 6), (4, 6)]])
    assert "hole count" in diff_reason(solid, holed)

test_areas_match_the_reviewed_values is the test that gives the whole fixture its authority: it compares the fixture against the numbers a person checked, so a corrupted fixture is caught independently of whether the transformation agrees with it.

Failure recovery

The golden file and the code disagree and nobody knows which is right. Go to the provenance note. If it names a control point or a published area, re-derive from that source and settle it in minutes. If it says nothing, the file was never evidence — establish an expected output properly now, and treat the intervening period as untested.

A golden file too large to review. Do not regenerate it; replace it. Extract the three or four shapes that actually exercise distinct behaviour, review those, and delete the rest. A six-shape reviewed fixture is worth more than a 40 000-feature snapshot.

A failure caused by a legitimate improvement. A better transformation may produce different — and better — output. Update the golden file and the provenance in the same commit, recording who re-verified it and against what. The --regenerate-golden warning exists to make that pairing hard to forget.

Frequently asked questions

Isn't a golden file generated from the code still useful?

Yes, as a regression detector — it tells you the behaviour changed, which is worth knowing. What it cannot do is tell you the behaviour was ever correct, so it should not be the only test of a transformation. The distinction matters most for reprojection, where a wrong transform produces output that is entirely self-consistent.

How large should the fixture be?

Small enough that one person can verify every expected value by hand, which in practice means three to eight shapes. Each shape should exist for a stated reason — a hole, a sliver, a clockwise ring, a flag lot — so the fixture doubles as documentation of what the transformation is contracted to handle.

Where do you get an expected output that is independent of the code?

Three usual sources: a surveyed control point whose coordinates are published in both systems, the county’s own published area or dimension figures for a real parcel, and hand arithmetic on deliberately round coordinates. Any of the three is enough; recording which one you used is what makes the fixture defensible later.

Should the comparison use exact equality?

Compare canonical forms with an explicit tolerance whose justification is recorded. Exact WKB equality fails on cosmetic differences that change nothing, and an unjustified tolerance drifts upward every time the suite goes red until it asserts nothing. Sharing the canonical function with production is what stops the test validating a different notion of equality than the pipeline uses.