Automating shapefile and GeoJSON exports from municipal portals

A county planning portal exposes a “Download zoning overlay” button that, behind the scenes, fires an asynchronous export job, stages a compressed Shapefile or GeoJSON archive, and hands back a one-time download token. You need an unattended job that does the same thing every night — fetch the export, confirm it is intact, reproject it, normalize its attributes, and swap it into your parcel fabric — without a missing .prj file or a truncated DBF column silently corrupting downstream spatial joins. This page answers that one narrow question: how do you turn a flaky, token-gated municipal export endpoint into a deterministic, recoverable ingestion step? It is a concrete application of GIS export sync workflows, focused on the exact moment a portal generates an on-demand archive and your automation has to trust it. The most frequent failure mode is the transition between legacy State Plane Shapefiles and WGS84 GeoJSON, where projection drift, attribute truncation, and invalid geometries break parcel matching with no exception at all.

Diagnosis: identifying where the export step silently corrupts data jump to heading

Municipal export endpoints behave like state machines, not synchronous data pipes, and they fail in three recognizable ways. Learn to spot all three before writing fetch code.

1. The export token expires or the archive arrives partial. A POST to the overlay endpoint returns 202 Accepted with a job token; you poll, download too early, and unzip a truncated file. The symptom is a zipfile.BadZipFile or a Shapefile missing its companions:

   Copy

   Traceback (most recent call last):
     File "/opt/pipeline/fetch.py", line 44, in fetch_municipal_export
       with zipfile.ZipFile(archive_path, "r") as z:
   zipfile.BadZipFile: File is not a zip file
   

2. Silent projection drift. The portal exports EPSG:2263 (NY State Plane, feet) or EPSG:26918 (NAD83 / UTM 18N), but the .prj is missing or carries malformed Well-Known Text. geopandas reads crs=None, and a later serialization step raises — or worse, writes coordinates in feet into a layer everything else assumes is WGS84:

   Copy

   pyproj.exceptions.CRSError: Invalid projection: unknown
   

3. Attribute truncation and type drift. Shapefiles cap field names at 10 characters and strings at 254. The same field arrives as "ZONE_CD": "R-1" from the DBF and "ZONE_CD": 1 from a later GeoJSON export. No exception fires; a downstream parcel-matching filter just returns zero rows.

Profile the archive before trusting it. List the Shapefile companions and read the declared CRS without loading the full geometry set, so you can fail fast on a structurally broken export:

import fiona
from pathlib import Path

REQUIRED_SIDECARS = {".shp", ".shx", ".dbf", ".prj"}

def inspect_export(workspace: Path) -> None:
    shp = next(workspace.glob("*.shp"), None)
    if shp is None:
        raise FileNotFoundError("No .shp in archive; portal may have exported GeoJSON only.")
    present = {p.suffix.lower() for p in workspace.glob(f"{shp.stem}.*")}
    missing = REQUIRED_SIDECARS - present
    if missing:
        raise ValueError(f"Shapefile incomplete; missing {sorted(missing)}.")
    with fiona.open(shp) as src:
        print(f"{shp.name}: crs={src.crs}, features={len(src)}, fields={list(src.schema['properties'])}")

A missing .prj in that output is your cue to halt before transformation, not after. This is the same fail-fast discipline that GIS export sync workflows apply at every staging boundary.

Step-by-step implementation jump to heading

Each step isolates exactly one concern. Compose them into the orchestrator at the end.

Step 1 — Poll the export token with bounded exponential backoff jump to heading

Treat the download phase as a transactional boundary. Poll the status endpoint with jittered, capped backoff so you stage the archive only once the job reports COMPLETED, and never hammer a portal hard enough to trip its WAF. Aggressive polling here is exactly what municipal API rate limit management exists to prevent.

import time
import zipfile
import requests
from pathlib import Path

def fetch_municipal_export(status_url: str, token: str, max_retries: int = 12) -> Path:
    workspace = Path(f"/tmp/muni_export_{token}")
    workspace.mkdir(parents=True, exist_ok=True)

    for attempt in range(max_retries):
        status_resp = requests.get(status_url, params={"token": token}, timeout=15)
        status_resp.raise_for_status()
        payload = status_resp.json()

        if payload.get("state") == "COMPLETED":
            archive_resp = requests.get(payload["download_url"], stream=True, timeout=60)
            archive_resp.raise_for_status()
            archive_path = workspace / "export.zip"
            with open(archive_path, "wb") as f:
                for chunk in archive_resp.iter_content(chunk_size=8192):
                    f.write(chunk)
            with zipfile.ZipFile(archive_path, "r") as z:
                z.extractall(workspace)
            return workspace

        # Jittered exponential backoff, capped to avoid WAF blocks.
        time.sleep(min(2 ** attempt + (attempt * 0.5), 30))

    raise RuntimeError("Export polling timed out; endpoint may be rate-limited or offline.")

Step 2 — Enforce and validate the CRS before any transformation jump to heading

Implicit CRS inference is an anti-pattern. Reject a layer with no declared projection, catch coordinates that exceed geographic bounds (the signature of projected meters or feet masquerading as degrees), and only then reproject. Strict, explicit projection handling is the cross-jurisdiction discipline detailed in CRS alignment strategies.

import geopandas as gpd
from shapely.validation import make_valid

def enforce_and_validate_crs(gdf: gpd.GeoDataFrame, expected_epsg: int = 4326) -> gpd.GeoDataFrame:
    if gdf.crs is None:
        raise ValueError("Source lacks .prj metadata; manual CRS assignment required.")

    bounds = gdf.total_bounds  # (minx, miny, maxx, maxy)
    if abs(bounds[0]) > 180 or abs(bounds[1]) > 90:
        raise ValueError("Coordinates exceed geographic bounds; verify CRS before transforming.")

    if gdf.crs.to_epsg() != expected_epsg:
        gdf = gdf.to_crs(epsg=expected_epsg)

    # Repair self-intersections and ring-orientation faults before serialization.
    gdf["geometry"] = gdf["geometry"].apply(make_valid)
    if not gdf["geometry"].is_valid.all():
        raise ValueError("Invalid geometries persist after repair; abort to protect spatial joins.")

    return gdf

This guard prevents the RuntimeError: Cannot transform geometry with unknown CRS that routinely halts unmonitored ingestion, and it stops a feet-based State Plane export from being written as if it were degrees.

Step 3 — Normalize attributes against a canonical schema jump to heading

Map the export’s columns onto a fixed canonical schema, cast every field to a known type, and replace nulls explicitly rather than letting NaN poison a dtype. This is the same contract enforced by attribute normalization, applied at the export boundary so type drift never reaches the parcel index.

CANONICAL_SCHEMA = {
    "ZONE_CD": str,
    "LAND_USE": str,
    "ACRES": float,
    "LAST_UPD": str,
}

def normalize_attributes(gdf: gpd.GeoDataFrame) -> gpd.GeoDataFrame:
    for col, dtype in CANONICAL_SCHEMA.items():
        if col in gdf.columns:
            gdf[col] = gdf[col].astype(dtype)
            if dtype is str:
                gdf[col] = gdf[col].str.strip().replace("", "UNKNOWN")
            elif dtype is float:
                gdf[col] = gdf[col].fillna(0.0)
        else:
            gdf[col] = "UNKNOWN" if dtype is str else 0.0  # absent column, explicit default
    return gdf

Step 4 — Serialize and emit a compliance manifest under a two-phase commit jump to heading

Never overwrite production in place. Stage the validated GeoJSON in a working directory, generate the manifest, and only swap pointers once everything passes. The manifest is the audit artifact that downstream schema validation & data quality checks consume.

import hashlib
import json
from datetime import datetime, timezone

def serialize_and_manifest(gdf: gpd.GeoDataFrame, token: str, staging_dir: Path) -> Path:
    staging_dir.mkdir(parents=True, exist_ok=True)
    ts = datetime.now(timezone.utc).strftime("%Y%m%d")
    geojson_path = staging_dir / f"zoning_{token}_{ts}.geojson"
    gdf.to_file(geojson_path, driver="GeoJSON")

    manifest = {
        "pipeline_id": token,
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "source_crs": gdf.crs.to_epsg(),
        "feature_count": len(gdf),
        "geometry_valid_count": int(gdf["geometry"].is_valid.sum()),
        "output_checksum": hashlib.sha256(geojson_path.read_bytes()).hexdigest(),
        "schema_version": "1.2.0",
    }
    (staging_dir / f"manifest_{token}.json").write_text(json.dumps(manifest, indent=2))
    return geojson_path

Step 5 — Compose the recoverable orchestrator jump to heading

Wire the steps into a single function whose every stage either advances cleanly or raises before touching production.

def run_zoning_sync_pipeline(status_url: str, token: str, staging_dir: Path) -> None:
    workspace = fetch_municipal_export(status_url, token)   # Phase 1: fetch & extract
    inspect_export(workspace)                               # fail fast on broken archives
    shp_path = next(workspace.glob("*.shp"))

    gdf = gpd.read_file(shp_path)                           # Phase 2: load & validate
    gdf = enforce_and_validate_crs(gdf, expected_epsg=4326)
    gdf = normalize_attributes(gdf)

    serialize_and_manifest(gdf, token, staging_dir)        # Phase 3: stage & manifest
    print(f"Sync staged at {staging_dir}; promote only after verification passes.")

Verification & testing jump to heading

Confirm the export actually round-tripped without silent loss before promoting it.

• Bounding box lands in jurisdiction. After reprojection, assert the transformed total_bounds falls inside the municipality’s known extent. A box centered off the coast of Africa (0, 0) means a CRS was assigned, not transformed.
• Geometry validity is total. assert gdf["geometry"].is_valid.all() — the manifest’s geometry_valid_count must equal feature_count. A shortfall means make_valid could not repair a topology and the layer is unsafe to join.
• Schema conforms exactly. assert set(CANONICAL_SCHEMA).issubset(gdf.columns) and check each dtype; an object column where you expect float64 means a string escaped Step 3.
• GeoJSON parses as RFC 7946. Reload the staged file with gpd.read_file and confirm the feature count matches; this catches encoding or driver faults that to_file swallowed. The serialized output must conform to RFC 7946: The GeoJSON Format, which mandates WGS84 (EPSG:4326).
• Checksum is deterministic. Re-running the same source export must reproduce the manifest output_checksum. Drift signals non-deterministic feature ordering — sort before serializing.

A quick reference for the projections you will meet most often at the export boundary:

Failure recovery jump to heading

When a stage fails mid-run, keep production untouched and stay reproducible.

• A failed sync leaves zero partial state. Because Step 4 stages to a working directory and the pointer swap is the last action, an exception anywhere upstream simply leaves the last known good layer live. Retain the raw archive and workspace for forensic review instead of deleting on error.
• Retry the fetch, abort the parse. A timed-out token or BadZipFile is transient — re-request a fresh export token through your error handling & retry logic. A CRS or schema exception is structural; do not retry it, quarantine the archive and emit a structured alert so a human reviews the municipal change.
• Quarantine breaking schema changes. When a portal drops ZONE_CD or renames a field, halt that one source, write the archive and its manifest to a review_required/ partition, and let sibling jurisdictions continue. Never let one municipality’s redesign abort the whole sync.
• Parallelize without losing isolation. When syncing many portals on a schedule, run each through async batch processing so a single slow endpoint never blocks the others, while each export keeps its own isolated workspace and two-phase commit.

Frequently asked questions jump to heading

Related jump to heading

• Parent topic: GIS Export Sync Workflows
• Section overview: Automated Feed Ingestion & GIS Data Parsing
• CRS Alignment Strategies — the projection discipline Step 2 enforces
• Attribute Normalization Rules — the canonical schema contract behind Step 3
• Schema Validation & Data Quality Checks — consuming the export manifest downstream
• Municipal API Rate Limit Management — polling without tripping a portal’s WAF
• Error Handling & Retry Logic — retry, quarantine, and alert patterns for failed fetches