Error Handling & Retry Logic for Municipal GIS Pipelines

Municipal GIS pipelines run against infrastructure that was never designed to be polled by machines. County APIs enforce undocumented rate limits, zoning PDFs change their encoding mid-cycle, ArcGIS REST endpoints time out under load, and coordinate reference systems drift between releases. In that environment transient failure is not the exception — it is the steady state, and a pipeline that treats every error the same way will either retry itself into an IP ban or silently write a corrupt parcel into the production map. The specific operational need this page addresses is deterministic recovery: knowing, for every class of failure, whether to retry, repair, quarantine, or roll back, and proving afterward that the decision was made correctly. It is one of the resilience layers of the Automated Feed Ingestion & GIS Data Parsing framework, and it wraps every other stage — acquisition, parsing, validation, and write — rather than living as a single step. Get it wrong and partial geometries, orphaned parcel records, and undetected compliance gaps propagate into PropTech analytics; get it right and a flaky county feed becomes a dependable, replayable source.

Prerequisites and operational context jump to heading

Retry logic is only safe once the surrounding pipeline gives it something deterministic to act on. Before the patterns below pay off, several things must already be in place:

• A staging boundary. Raw payloads must land in versioned object storage or a raw_features table before any transformation runs. Retries then replay from staging, never from the live municipal endpoint, so re-processing a failed batch never re-hammers a county server that is already struggling.
• A known target projection. Geometry repair and validity checks only make sense against a single working CRS. That depends on the CRS alignment strategies you standardise on — typically EPSG:4326 for storage — so a “repaired” polygon is not silently reprojected into the ocean.
• A normalization contract. The keys a write touches (parcel_id, zoning_code, jurisdiction) must already conform to your attribute normalization rules; otherwise a retry faithfully re-writes the same inconsistent zoning code it failed on the first time.
• A rate-limit budget. Backoff parameters are derived from observed throttling behaviour, governed by municipal API rate limit management, not guessed. A retry strategy that ignores the source’s tolerated request rate amplifies the very outage it is trying to survive.
• An idempotent write path. Every database write must be replay-safe before any retry is enabled. Retrying a non-idempotent insert is how a single network timeout becomes a duplicate parcel.

If any of these are missing, fix them first. Retries amplify whatever you feed them — including malformed input, unbounded request rates, and non-idempotent writes.

Architecture: tiered failure boundaries jump to heading

The single most important design decision is to scope retries to specific pipeline stages rather than wrapping the whole pipeline in one blanket try/except. A global retry on a malformed GeoJSON wastes compute, masks a real data-quality problem, and can loop forever on an error that will never succeed. Instead, classify every failure into one of four boundaries, each with its own decision:

• Network / transport layer — HTTP 429, 502, 503, 504, TLS resets, socket timeouts. These are genuinely transient and safe to retry with exponential backoff and jitter.
• Parsing / extraction layer — malformed XML, broken PDF encodings, missing spatial references coming out of the PDF & HTML scraping pipelines. These need validation gates and fallback parsers, not retries; the same bytes will fail the same way.
• Spatial / topology layer — invalid polygons, self-intersections, CRS mismatches. These call for geometric repair first, then dead-letter routing if repair fails.
• Compliance / write layer — constraint violations, duplicate zoning IDs, transaction rollbacks. These demand idempotent upserts and immutable audit logging, never a blind retry.

The decision a failure receives — retry, repair, quarantine, or roll back — is a function of which boundary it belongs to, and the retry policy itself should be declarative, state-aware, and bounded by both a maximum attempt count and a total wall-clock window. The flow through these boundaries is shown below.

This separation is what lets one county’s broken PDF coexist with another county’s 429 storm in the same run: each error is handled where it belongs, and a failure in one band never cascades into corruption in the next.

Production implementation jump to heading

Transport retries: exponential backoff with jitter jump to heading

Municipal servers throttle bulk requests aggressively, and naive linear retries trigger IP bans or queue starvation. Use tenacity for declarative retry decorators that combine exponential backoff, jitter, and a custom retry predicate so that only genuinely transient transport errors are retried.

import logging

import requests
import tenacity
from requests.exceptions import ConnectionError, HTTPError, Timeout

logger = logging.getLogger(__name__)

# Only these HTTP statuses represent a transient server-side condition.
RETRYABLE_STATUS = {429, 500, 502, 503, 504}


def is_retryable_error(exc: BaseException) -> bool:
    """Retry transport-level failures only; never retry a 4xx client error."""
    if isinstance(exc, (ConnectionError, Timeout)):
        return True
    if isinstance(exc, HTTPError) and exc.response is not None:
        return exc.response.status_code in RETRYABLE_STATUS
    return False


@tenacity.retry(
    retry=tenacity.retry_if_exception(is_retryable_error),
    # Per-attempt jitter prevents a thundering herd when workers restart together.
    wait=tenacity.wait_random_exponential(multiplier=2, min=1, max=30),
    # Bound by BOTH attempt count and total elapsed time.
    stop=tenacity.stop_after_attempt(5) | tenacity.stop_after_delay(120),
    before_sleep=tenacity.before_sleep_log(logger, logging.WARNING),
    reraise=True,
)
def fetch_zoning_parcel(parcel_id: str, endpoint: str, headers: dict) -> dict:
    """Fetch one parcel record with bounded, jittered transport retries."""
    resp = requests.get(f"{endpoint}/{parcel_id}", headers=headers, timeout=15)
    # Honour an explicit cooldown before tenacity computes its own backoff.
    if resp.status_code == 429:
        retry_after = resp.headers.get("Retry-After")
        if retry_after:
            logger.warning("429 on %s; server requested %ss cooldown", parcel_id, retry_after)
    resp.raise_for_status()
    return resp.json()

wait_random_exponential adds per-attempt jitter, which prevents the thundering-herd problem when many workers back off in lockstep and then retry simultaneously. The compound stop enforces both an attempt ceiling and a wall-clock ceiling, so a prolonged county outage drains the queue gracefully instead of pinning workers for hours. Because is_retryable_error rejects 4xx client errors, a 404 on a deleted parcel fails fast and falls through to quarantine rather than burning five attempts.

Spatial repair and dead-letter routing jump to heading

A successful HTTP fetch is worthless if the payload contains invalid geometry, so spatial validation runs immediately after parsing and before any write. Use shapely to enforce OGC Simple Features validity, attempt an automated repair, and route anything unrepairable to a quarantine table rather than retrying it.

import logging

import geopandas as gpd
from shapely.geometry import shape
from shapely.validation import make_valid

logger = logging.getLogger(__name__)


class DeadLetter(Exception):
    """Raised when a record cannot be repaired and must be quarantined."""


def validate_and_repair_geometry(
    feature: dict, target_crs: str = "EPSG:4326"
) -> gpd.GeoDataFrame:
    """Validate, repair, and CRS-tag one feature, or route it to dead-letter."""
    fid = feature.get("id") or feature.get("properties", {}).get("parcel_id")
    try:
        geom = shape(feature["geometry"])
        if not geom.is_valid:
            geom = make_valid(geom)  # fixes bowties, self-intersections

        # make_valid can emit a GeometryCollection (e.g. polygon + stray line).
        # Keep only polygonal parts; a non-polygonal result is degenerate here.
        if geom.geom_type == "GeometryCollection":
            polys = [g for g in geom.geoms if g.geom_type in ("Polygon", "MultiPolygon")]
            if not polys:
                raise DeadLetter(f"No polygonal part after repair for {fid}")
            geom = polys[0] if len(polys) == 1 else gpd.GeoSeries(polys).union_all()

        if geom.is_empty or geom.area == 0:
            raise DeadLetter(f"Degenerate geometry after repair for {fid}")

        return gpd.GeoDataFrame([feature], geometry=[geom], crs=target_crs)
    except DeadLetter:
        raise
    except Exception as exc:  # malformed coords, unsupported type
        logger.error("Unrepairable geometry for feature %s: %s", fid, exc)
        raise DeadLetter(str(exc)) from exc

When make_valid cannot produce a usable polygon, the record is sent to a quarantine table for human-in-the-loop review or automated correction downstream in the GIS export sync workflows; it is never retried, because the same bytes will always fail the same way. Filtering the GeometryCollection to its polygonal parts is the gotcha most pipelines miss — make_valid legitimately returns mixed collections, and writing one to a Polygon/MultiPolygon column raises a type error far downstream.

Idempotent writes and transactional safety jump to heading

Database writes must survive partial failures so that retrying a network timeout never duplicates a record. Implement idempotent upserts with PostgreSQL ON CONFLICT, wrapped in a transaction, and treat a constraint violation as a signal to audit rather than to retry blindly.

import logging

from sqlalchemy import text
from sqlalchemy.exc import IntegrityError

logger = logging.getLogger(__name__)

_UPSERT = text(
    """
    INSERT INTO zoning_parcels (parcel_id, zoning_code, geometry, source_hash, updated_at)
    VALUES (:parcel_id, :zoning_code, ST_GeomFromText(:geom_wkt, 4326), :source_hash, NOW())
    ON CONFLICT (parcel_id) DO UPDATE SET
        zoning_code = EXCLUDED.zoning_code,
        geometry    = EXCLUDED.geometry,
        source_hash = EXCLUDED.source_hash,
        updated_at  = NOW()
    WHERE zoning_parcels.source_hash IS DISTINCT FROM EXCLUDED.source_hash
    """
)


def upsert_zoning_record(engine, record: dict) -> bool:
    """Idempotent, replay-safe upsert. Returns False on a logged constraint error."""
    try:
        with engine.begin() as conn:  # commits on success, rolls back on raise
            conn.execute(_UPSERT, record)
        return True
    except IntegrityError as exc:
        # A constraint violation is a data problem, not a transient one — never retry.
        logger.warning("Constraint violation for parcel %s: %s", record["parcel_id"], exc)
        return False

The source_hash guard means an unchanged record is a no-op even on replay, so re-running a completed batch produces neither duplicates nor spurious updated_at churn in the audit trail. Pairing the upsert with engine.begin() guarantees that a mid-write failure rolls the whole statement back rather than leaving a half-written parcel. Constraint violations are caught and logged, not retried — a duplicate zoning ID is a problem upstream in the data, and looping on it only delays detection.

Edge cases and gotchas jump to heading

Municipal feeds break in specific, recurring ways. Plan for these before they reach production:

• 429 storms across one jurisdiction. A global retry budget still means many simultaneous requests to the same county. Pair backoff with a per-host token bucket so one slow municipality cannot starve every other source’s budget.
• Retry-After in HTTP-date form. Some gateways return Retry-After as an absolute date, not seconds. Parse both forms before feeding the value to your sleep, or you will either retry instantly or block for a day.
• Topology exceptions on repair. make_valid can raise on coordinates with NaN/Inf values produced by a bad reprojection. Catch it explicitly and dead-letter the record rather than letting it abort the batch.
• Datum drift that passes validation. A feed may declare EPSG:4326 but ship state-plane coordinates. The geometry is “valid” and the upsert succeeds, but the parcel lands in the wrong hemisphere. Sanity-check that transformed bounds fall inside the jurisdiction’s expected bounding box before writing.
• Schema version mismatches. When a county renames zoning to zone_class between releases, record.get("zoning") silently returns None and the upsert writes a null zoning code. Assert on required keys and route misses to quarantine — this is where schema validation & data quality checks belong.
• Retries that mask poison messages. A record that fails deterministically (bad geometry, missing key) will fail every retry. Without a dead-letter path it loops forever or blocks the queue. Every non-transient failure must have a terminal quarantine state.
• Unbounded total time. Attempt-count limits alone are not enough — five attempts with a 30-second backoff can still pin a worker for minutes. Always cap total elapsed time as well, as the stop_after_delay above does.

Retry strategy reference jump to heading

Different failure classes deserve different behaviour. Treating them uniformly either wastes compute or hides real data problems.

Integration points jump to heading

Error handling is a cross-cutting layer, defined by the stages it wraps. Upstream, it consumes the transport failures and malformed payloads surfaced by the PDF & HTML scraping pipelines and the concurrent fetch stage of async batch processing, where per-stage retry boundaries live inside each worker rather than around the whole loop. Downstream, when a source has been failing for hours and retries are exhausted, the decision is no longer “retry” but “substitute”: the exhausted partition hands off to fallback routing logic, which fills the gap from a secondary source or a last-known-good snapshot rather than leaving a hole in the map. The contract across both directions is the quarantine record shape and the retry-exhausted signal — same reason codes, same hashes — so that recovery can be reasoned about end to end.

Compliance and audit artifacts jump to heading

For PropTech underwriting and regulatory review, a run is not finished when the data lands — it is finished when the failure handling is provable. Every run should emit:

• A retry ledger: per-request attempt counts, final disposition, and the backoff actually applied, so a reviewer can distinguish a clean fetch from one that succeeded only after four 429s.
• An immutable quarantine ledger: every dead-lettered record with its reason code (invalid geometry, schema miss, datum-bounds failure), retained for audit and never silently purged.
• Per-stage failure metrics: retry_attempts, dead_letter_count, and spatial_repair_rate per municipality, feeding a circuit breaker that halts ingestion when a source’s failure rate exceeds a threshold (for example, >15% over five minutes) during a county system migration.
• Source content hashes: the source_hash written with each upsert links every parcel back to the exact payload and code that produced it, making the entire chain replayable and tamper-evident.

These artifacts pair with downstream schema validation & data quality checks to form a continuous, defensible chain from raw municipal publication to underwriting-grade dataset — every zoning change, parcel split, and boundary adjustment captured, validated, and traceable.

FAQ jump to heading

Related jump to heading

• Async batch processing — where per-stage retry boundaries live inside each concurrent worker
• Municipal API rate limit management — sets the backoff budget this layer must respect
• PDF & HTML scraping pipelines — the source of most parse-layer failures
• GIS export sync workflows — downstream consumer of validated, repaired records
• Fallback routing logic — what happens when retries are exhausted and a source must be substituted
• Schema validation & data quality checks — catches schema-layer failures before they reach the write

Up: Automated Feed Ingestion & GIS Data Parsing