Serving last-known-good snapshots when a municipal feed goes dark

Pima County’s zoning REST endpoint has been returning 503s for three days after a portal migration, and your consumers still expect an answer every time they ask “what is the zoning on this parcel?” You have two bad options if you do nothing deliberate: crash the query and hand PropTech users an error where they expect a district, or let the pipeline write an empty result and quietly blank out every parcel the dead feed used to cover. Both are worse than the honest third path — keep serving the last data you know was good, but stamp it so nobody mistakes three-day-old zoning for a live read. This guide sits within fallback routing logic in the wider municipal zoning data architecture and compliance frameworks area, and it walks the full lifecycle: detecting that a feed has genuinely gone dark, promoting the last validated snapshot, tagging every response with its as-of date and staleness, degrading to a coarser source when honesty demands it, and auto-recovering the moment the feed comes back.

Diagnosis: a dark feed is not an empty result jump to heading

The dangerous ambiguity is that a healthy fetch of a genuinely-empty area and a broken fetch of a populated area can both return zero rows. If you treat “zero rows” as truth, a transient outage silently deletes coverage. Distinguish the two before you decide anything.

# Two responses that look identical to a naive `if not rows:` check
outage   = {"http": 503, "rows": [], "elapsed_s": 0.2}   # feed is DOWN
genuinely_empty = {"http": 200, "rows": [], "elapsed_s": 1.1}  # area truly has 0 parcels

# The trap: this blanks out a whole county during an outage.
def bad_ingest(resp):
    if not resp["rows"]:
        overwrite_target(resp["rows"])   # writes emptiness as if it were fact

Three signals separate a dark feed from an honest empty result:

1. Transport-level failure. A non-2xx status, a connection reset, or a timeout is an outage, full stop — it is never evidence about the data and must never overwrite the target. A 200 with zero rows is at least a claim about reality and can be evaluated on its merits.
2. Volume collapse against baseline. A feed that returned 41,000 parcels yesterday and 3 today has not lost 40,997 parcels overnight; it is half-migrated or truncating. Compare row counts to a rolling baseline and treat a sharp collapse as a soft outage even when the HTTP status is 200.
3. Staleness of the upstream itself. Many portals expose a last_edited_date or layer editingInfo. If that timestamp has not advanced and your own copy already reflects it, the feed is merely quiet, not dark — do not confuse “no new edits” with “no data.”

Only after classifying the response do you decide whether to accept a write, hold the last snapshot, or degrade. The rest of this guide implements that decision.

Step-by-step implementation jump to heading

Each step is one concern. Compose them in the orchestrator at the end.

Step 1 — Health-check the feed and detect staleness jump to heading

Run the classification from the diagnosis as an explicit function that returns a verdict, never a bare boolean. Feed it the HTTP result, the returned row count, and a rolling baseline so a volume collapse is caught even on a 200.

from dataclasses import dataclass

@dataclass
class FeedVerdict:
    healthy: bool
    reason: str

def classify_feed(http_status: int, rows: int, baseline: int,
                  connection_error: bool = False) -> FeedVerdict:
    """Separate a dark feed from a genuinely-empty result."""
    if connection_error or http_status >= 500:
        return FeedVerdict(False, f"transport_failure:{http_status}")
    if http_status != 200:
        return FeedVerdict(False, f"unexpected_status:{http_status}")
    # A collapse to under 10% of a non-trivial baseline is a soft outage.
    if baseline >= 100 and rows < baseline * 0.10:
        return FeedVerdict(False, f"volume_collapse:{rows}/{baseline}")
    return FeedVerdict(True, "ok")

Step 2 — Promote the last validated snapshot jump to heading

When the verdict is unhealthy, do not write. Instead resolve the most recent snapshot that passed validation and mark it as the active last-known-good. Keeping snapshots as immutable, timestamped rows is the same pattern behind temporal versioning and snapshots on the analysis side.

import psycopg
from datetime import datetime, timezone

def promote_last_known_good(conn, jurisdiction: str) -> dict | None:
    """Pin the newest validated snapshot as the served copy; never overwrite."""
    with conn.cursor(row_factory=psycopg.rows.dict_row) as cur:
        cur.execute(
            """
            SELECT snapshot_id, as_of, row_count
            FROM zoning_snapshot
            WHERE jurisdiction = %s AND validation_passed IS TRUE
            ORDER BY as_of DESC
            LIMIT 1
            """,
            (jurisdiction,),
        )
        snap = cur.fetchone()
        if snap is None:
            return None  # cold start with no good snapshot -> caller must degrade
        cur.execute(
            "UPDATE serving_pointer SET snapshot_id = %s, promoted_at = %s "
            "WHERE jurisdiction = %s",
            (snap["snapshot_id"], datetime.now(timezone.utc), jurisdiction),
        )
    conn.commit()
    return snap

Step 3 — Tag every response with as-of and staleness jump to heading

The snapshot alone is not enough; the label is what keeps you honest. Every served record carries the snapshot’s as-of date, a computed staleness, and an explicit fresh flag, so a consumer can decide whether three-day-old zoning is acceptable for their use.

from datetime import datetime, timezone

def tag_response(record: dict, as_of: datetime,
                 fresh_threshold_hours: float = 26.0) -> dict:
    """Stamp provenance so stale data can never masquerade as live."""
    age_h = (datetime.now(timezone.utc) - as_of).total_seconds() / 3600
    return {
        **record,
        "as_of": as_of.isoformat(),
        "staleness_hours": round(age_h, 1),
        "fresh": age_h <= fresh_threshold_hours,
        "source_status": "live" if age_h <= fresh_threshold_hours else "last_known_good",
    }

Step 4 — Degrade gracefully to a coarser source jump to heading

Past a staleness ceiling, a parcel-precise answer that is a week old can be more misleading than a coarser but current one. Fall back to a lower-resolution source — a county-wide generalized zoning layer or the parent jurisdiction — and label the drop in precision explicitly rather than pretending the parcel-level answer still holds.

def serve_with_degradation(record: dict | None, as_of, staleness_ceiling_h: float,
                           coarse_lookup) -> dict:
    """Beyond the staleness ceiling, answer from a coarser source, labelled."""
    if record is not None:
        tagged = tag_response(record, as_of)
        if tagged["staleness_hours"] <= staleness_ceiling_h:
            return tagged
    # Too stale (or no snapshot at all): drop to a coarser, current source.
    coarse = coarse_lookup(record["parcel_id"] if record else None)
    return {
        **coarse,
        "source_status": "degraded_coarse",
        "precision": "jurisdiction",   # not parcel-level
        "fresh": True,
    }

Step 5 — Auto-recover when the feed returns jump to heading

Recovery must be automatic and validated. A watcher retries the dark feed on a backoff; the first fetch that passes classification and schema validation promotes a fresh snapshot and flips the pointer back to live. Use the same backoff-with-jitter and dead-letter discipline from error handling and retry logic so probes don’t hammer a recovering portal.

def attempt_recovery(conn, jurisdiction: str, fetch_fn, validate_fn) -> bool:
    """One recovery probe: promote a fresh snapshot only if it fully validates."""
    try:
        resp = fetch_fn(jurisdiction)  # applies backoff + jitter internally
    except Exception:
        return False  # still dark; watcher will retry on the next tick
    verdict = classify_feed(resp["http"], len(resp["rows"]),
                            baseline=resp.get("baseline", 0))
    if not verdict.healthy:
        return False
    if not validate_fn(resp["rows"]):
        return False  # feed answered but data is malformed -> stay on last-known-good
    _write_new_snapshot(conn, jurisdiction, resp["rows"])  # flips pointer to live
    return True

Verification & testing jump to heading

Prove that stale data is always labelled and that recovery is clean.

• Outage never overwrites. Feed a 503 and assert the target row count is unchanged and the serving pointer still references the prior snapshot. A shrink means the outage leaked into a write.
• Empty is distinguished from dark. Assert a genuine 200 with zero rows over a zero baseline is accepted, while a 200 collapsing from 41,000 to 3 is classified as a soft outage.
• Every stale response is labelled. Snapshot the serving output during a simulated outage and assert every record has source_status == "last_known_good" and a non-zero staleness_hours. No record may report fresh: true while the feed is down.
• Degradation trips at the ceiling. Advance a synthetic clock past the staleness ceiling and assert responses switch to degraded_coarse with precision: "jurisdiction".
• Recovery is validated, not just reachable. A feed that returns malformed rows must stay on last-known-good; only a fetch that passes both classification and schema validation flips the pointer back to live.

Failure recovery jump to heading

The fallback machinery has its own failure modes, and each needs a deliberate answer.

• Cold start with no good snapshot. If promote_last_known_good returns None, there is nothing to hold. Do not serve emptiness — degrade immediately to the coarser source and surface an explicit “no baseline available” status so consumers know coverage is provisional.
• Snapshot store unavailable. If the snapshot database itself is unreachable, serve the last in-memory copy with a hard staleness cap and refuse to promote anything until the store returns, so you never pin a pointer you cannot later audit.
• Recovery flapping. A feed oscillating between 200 and 503 should not flip the pointer on every tick. Require N consecutive healthy, validated fetches before promoting, and log each flip so an alert fires when flap frequency crosses a threshold.
• Silent lingering staleness. A feed that never recovers must not fade into the background. Emit staleness as a per-jurisdiction metric and page when any served jurisdiction crosses its ceiling, so a dark Pima County feed becomes a ticket rather than a quietly rotting layer.

Frequently asked questions jump to heading

Related jump to heading

• Parent topic: Fallback Routing Logic
• Section overview: Municipal Zoning Data Architecture & Compliance Frameworks
• Error Handling & Retry Logic — backoff, jitter, and dead-letter probes for a recovering feed
• Temporal Versioning & Snapshots — immutable as-of snapshots the last-known-good copy resolves against