Designing fallback routing for missing municipal data feeds

Your nightly zoning sync wakes up, hits the county WFS endpoint it has used for two years, and gets back an HTTP 200 carrying an empty FeatureCollection — no error, no exception, just zero polygons silently overwriting yesterday’s parcels. This page answers one narrow question: when a primary municipal feed vanishes or degrades mid-run, how do you route to a secondary source without corrupting compliance state or introducing geometric drift? The pattern here is a concrete application of fallback routing logic, tuned for the exact moment a feed stops being trustworthy. The goal is a router that detects degradation deterministically, fails over through a fixed source hierarchy, re-validates schema and projection at each tier boundary, and emits an audit record for every switch.

Diagnosis: identifying a degraded feed before it corrupts state jump to heading

A broken municipal feed almost never announces itself with a clean 503 Service Unavailable. It degrades in ways a naive response.raise_for_status() waves straight through. There are four signatures worth instrumenting for, because each one needs a different trip condition.

1. Empty-but-successful payloads. The endpoint returns 200 OK with {"type": "FeatureCollection", "features": []}. A downstream gpd.read_file() succeeds, the spatial join produces zero matches, and the run commits an empty zoning layer over good data. This surfaces later as a ValueError: No features found or a silent coverage gap in the next analyst’s map.
2. Connection and read timeouts. A throttled or overloaded portal stalls the socket. Without explicit (connect, read) timeouts you get requests.exceptions.ReadTimeout — or worse, a hung worker that exhausts the thread pool and stalls the whole pipeline.
3. Silent schema mutation. The municipality renames ZONE_CODE to zoning_id, or flips effective_date from ISO-8601 to a Unix epoch integer. The fetch succeeds, but the attribute join drops every row, raising KeyError: 'ZONE_CODE' or quietly populating nulls into your compliance model.
4. Topology corruption. The feed delivers self-intersecting multipart polygons that pass JSON parsing but fail the spatial stage with shapely.errors.TopologicalError or a GEOS non-noded intersection during overlay.

Reproduce the empty-payload case first, because it is the most dangerous and the easiest to miss:

import requests

resp = requests.get(PRIMARY_URL, timeout=(3.05, 15))
payload = resp.json()
feature_count = len(payload.get("features", []))
print(f"status={resp.status_code} features={feature_count} "
      f"last_modified={resp.headers.get('Last-Modified')}")
# status=200 features=0  -> a 200 is NOT proof of a usable feed

The lesson the diagnosis enforces: health is not an HTTP status code. A feed is healthy only if it returns a minimum geometry count, the expected schema, and a Last-Modified newer than your last good snapshot. Everything below treats those three conditions as the trip wire.

Step-by-step implementation jump to heading

Each step below isolates exactly one concern: defining the hierarchy, deciding health, breaking the circuit, normalizing the fallback, and recording the switch.

Step 1 — Declare a deterministic source hierarchy jump to heading

Resolve sources in strict, configuration-driven priority order so failover is reproducible and never depends on which thread happened to fail first. Four tiers cover the realistic municipal landscape.

from dataclasses import dataclass

@dataclass(frozen=True)
class FeedTier:
    name: str          # "primary" | "secondary" | "cache" | "archive"
    url: str
    min_features: int  # below this, treat the payload as a failure
    max_age_hours: int # Last-Modified older than this is stale

TIERS = [
    FeedTier("primary",   "https://gis.county.gov/wfs/zoning", 50, 48),
    FeedTier("secondary", "https://state-clearinghouse.gov/zoning.geojson", 50, 168),
    FeedTier("cache",     "file:///snapshots/zoning_last_good.parquet", 50, 8760),
    FeedTier("archive",   "postgis://archive/zoning_history", 1, 87600),
]

The primary tier is the live municipal API; the secondary is a state or regional clearinghouse on a weekly cadence; the cache tier is your own last-known-good snapshot pinned by checksum; the archive is a reconstruction from prior ingests for catastrophic outages. The same tier hierarchy must respect the underlying municipal data structures so a flattened cache snapshot and a nested live FeatureCollection reconcile to one internal model.

Step 2 — Make the health decision explicit jump to heading

Wrap every fetch in a single predicate that returns a usable frame or None. No partial successes, no exceptions leaking to the caller.

import geopandas as gpd
import requests
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime

def fetch_tier(tier: FeedTier) -> gpd.GeoDataFrame | None:
    try:
        gdf = gpd.read_file(tier.url)            # works for WFS, GeoJSON, parquet, PostGIS
    except Exception as exc:                       # noqa: BLE001 - any read failure trips the tier
        log.warning("tier=%s read failed: %s", tier.name, exc)
        return None

    if len(gdf) < tier.min_features:              # empty-but-successful guard
        log.warning("tier=%s below floor: %d < %d", tier.name, len(gdf), tier.min_features)
        return None

    return gdf

The min_features floor is what catches the empty FeatureCollection that a status check misses. Tie staleness checks to the Last-Modified header where the transport exposes it, and skip a tier whose freshest record predates your last good snapshot.

Step 3 — Gate the circuit with a sliding failure window jump to heading

A single hiccup should not abandon the primary feed, but a sustained failure rate should. Track outcomes in a bounded window and open the circuit when the failure ratio crosses a threshold, so you stop hammering a dead endpoint and fail straight to the secondary.

from collections import deque

class CircuitBreaker:
    def __init__(self, window: int = 20, trip_ratio: float = 0.4):
        self.window = deque(maxlen=window)   # True = success, False = failure
        self.trip_ratio = trip_ratio

    def record(self, ok: bool) -> None:
        self.window.append(ok)

    def is_open(self) -> bool:
        if len(self.window) < self.window.maxlen:
            return False
        failures = self.window.count(False)
        return (failures / len(self.window)) >= self.trip_ratio

Pair the breaker with bounded exponential backoff and jitter on transient primary errors so retries never synchronize into a thundering herd — the same discipline detailed under error handling & retry logic, here scoped to the decision of when to stop trying the primary at all. For pre-emptive throttling against quota-limited portals, lean on municipal API rate limit management so the breaker rarely has to trip in the first place.

Step 4 — Re-validate schema and CRS at every tier boundary jump to heading

This is the step most fallback designs skip, and it is the one that quietly corrupts compliance models. A secondary or cache source is not schema-compatible by assumption — promote it to production only after it clears the same gate the primary does.

from pyproj import CRS

TARGET_CRS = CRS.from_epsg(4326)
REQUIRED = {"parcel_id", "zone_class", "effective_date"}

def conform(gdf: gpd.GeoDataFrame, tier: FeedTier) -> gpd.GeoDataFrame:
    missing = REQUIRED - set(gdf.columns)
    if missing:
        raise ValueError(f"tier={tier.name} schema drift, missing {missing}")

    if gdf.crs is None:
        gdf = gdf.set_crs(TARGET_CRS)            # never assume; only set when truly absent
    elif gdf.crs != TARGET_CRS:
        gdf = gdf.to_crs(TARGET_CRS)             # reproject mismatched datums

    invalid = ~gdf.geometry.is_valid
    if invalid.any():
        from shapely.validation import make_valid
        gdf.loc[invalid, "geometry"] = gdf.loc[invalid, "geometry"].apply(make_valid)

    return gdf

The schema contract here is deliberately strict; reconciling renamed columns across jurisdictions is the job of zoning taxonomy mapping upstream, while the projection logic is the inline version of the cross-jurisdiction CRS alignment strategies you apply at scale. The manifest this gate produces is exactly what feeds downstream schema validation & data quality checks.

Step 5 — Resolve the active source and stamp provenance jump to heading

Walk the tiers in order, skipping the primary when the breaker is open, and tag every record with the tier it came from so no fallback data is ever indistinguishable from authoritative data.

import hashlib, json

def resolve_feed(breaker: CircuitBreaker) -> tuple[gpd.GeoDataFrame, dict]:
    for tier in TIERS:
        if tier.name == "primary" and breaker.is_open():
            log.info("circuit open, skipping primary")
            continue

        gdf = fetch_tier(tier)
        ok = gdf is not None
        if tier.name == "primary":
            breaker.record(ok)
        if not ok:
            continue

        gdf = conform(gdf, tier)
        raw = gdf.to_json().encode()
        manifest = {
            "source_tier": tier.name,
            "source_uri": tier.url,
            "feature_count": len(gdf),
            "snapshot_hash": hashlib.sha256(raw).hexdigest()[:16],
            "ingestion_timestamp": datetime.now(timezone.utc).isoformat(),
            "confidence_score": 1.0 if tier.name == "primary" else 0.7,
        }
        gdf["source_tier"] = tier.name
        return gdf, manifest

    raise RuntimeError("all feed tiers exhausted")

Archive-tier records get a reduced confidence_score so analysts and underwriting models can flag reconstructed boundaries for manual review rather than treating them as ground truth.

Step 6 — Emit an immutable audit record on every switch jump to heading

Fallback does not excuse a compliance gap; it documents one. Append a structured event to immutable storage whenever the active tier changes, so a regulator can reconstruct exactly which source served a given boundary on a given day.

def emit_audit(manifest: dict, previous_tier: str | None) -> None:
    event = {
        **manifest,
        "event": "feed_tier_switch",
        "previous_tier": previous_tier,
        "schema_drift_detected": False,
        "geometry_count_delta": None,  # fill from last good snapshot count
    }
    with open("/audit/feed_switches.ndjson", "a") as fh:  # append-only ledger
        fh.write(json.dumps(event) + "\n")

Route the same event to your incident channel so a tier switch surfaces in real time, not at the next quarterly audit.

Verification & testing jump to heading

Confirm the router actually fails over and that fallback data is analytically equivalent to primary data.

• Simulated outage. Point the primary url at a mock returning an empty FeatureCollection, then a ReadTimeout, then a renamed-column payload. Each must produce a source_tier=secondary result and exactly one feed_tier_switch audit line — never an empty commit.
• Breaker trips and recovers. Feed the breaker 8 failures in a 20-event window (40%) and assert is_open() flips to True; feed successes back in and assert it closes once the window clears.
• Schema gate holds. Assert conform() raises on a payload missing zone_class rather than committing nulls — a silent null fill is the exact failure this gate exists to stop.
• Projection integrity. After failover to a state-plane secondary, assert gdf.crs == TARGET_CRS and that a known parcel centroid lands within tolerance of its primary-feed coordinate. A datum mismatch shows up as a tens-of-metres shift.
• Deterministic checksums. Re-running against the same cache snapshot must reproduce an identical snapshot_hash; drift signals non-deterministic row ordering — pin the sort before hashing.

A quick reference for setting the trip conditions per tier:

Failure recovery jump to heading

When the router itself runs out of road, fail loud and reproducibly rather than committing garbage.

• Exhaustion is a hard stop, not an empty write. If resolve_feed raises RuntimeError: all feed tiers exhausted, hold the previous good layer in place and emit a PIPELINE_HALTED event. Never let an all-tiers-down run truncate the live zoning table.
• Quarantine schema-drift payloads. A tier that fails the conform() gate writes its raw response to a drift_quarantine/ partition with the offending column diff, so you can patch the taxonomy map and replay only that source.
• Pin and roll back by checksum. Because every snapshot carries a snapshot_hash, recovery is a lookup: restore the last manifest whose source_tier == "primary" and re-point the live layer at its pinned snapshot.
• Half-open probe before full restore. When the primary returns, send a single canary fetch through the breaker in a half-open state; promote it back to primary only after one clean, schema-valid, above-floor response — otherwise it stays open and you keep serving the secondary.

Frequently asked questions jump to heading

Related jump to heading

• Parent topic: Fallback Routing Logic
• Section overview: Municipal Zoning Data Architecture & Compliance Frameworks
• Municipal Data Structures — reconciling nested live feeds with flattened cache snapshots
• Schema Validation & Data Quality Checks — the gate every promoted tier must clear
• CRS Alignment Strategies — reprojecting fallback sources to a canonical CRS
• Error Handling & Retry Logic — backoff, dead-letter, and checkpoint patterns behind the breaker
• Async Batch Processing — streaming large fallback snapshots without exhausting memory