Municipal API Rate Limit Management

A county GIS server that tolerated 60 requests a minute on Tuesday starts returning 429 Too Many Requests at request 12 on Wednesday, because a budget hearing put the planning department’s portal under load and an upstream gateway tightened its quota without notice. Municipal endpoints rarely publish consistent rate-limit documentation, yet they enforce quotas that silently break automated zoning-change tracking the moment traffic crosses an invisible line. For PropTech teams and Python automation engineers, rate limiting is not an edge case to bolt on after launch — it is a core architectural constraint that determines whether a refresh completes or strands half a jurisdiction in a stale state. This page sits inside the Automated Feed Ingestion & GIS Data Parsing framework and defines the request-pacing contract that every fetch in the pipeline must respect: deterministic outbound throttling, header-aware backoff, per-jurisdiction isolation, and graceful fallback when a quota is exhausted.

Prerequisites and operational context jump to heading

Rate-limit management is meaningful only when the surrounding pipeline already enforces a few invariants. Build these first, or the limiter will pace traffic against a moving target:

• A staging layer between fetch and database. Requests must be replayable. When raw payloads land in versioned object storage before transformation, a re-run after a throttle event reads from staging instead of re-hammering the municipal server. The limiter governs only the acquisition boundary, not reprocessing.
• A working projection contract. The fetch layer decides what to request — and stripping or deferring geometry is one of the largest levers on quota consumption. That decision depends on the CRS alignment strategies you have standardised on, because re-fetching coordinates to reproject them later doubles request volume.
• A concurrency model that asks permission. Every concurrent fetch in async batch processing must consult the limiter before issuing a request. The semaphore that bounds in-flight requests and the token bucket that bounds request rate are two different gates; both have to be honoured.
• A normalization contract for responses. Whatever the limiter lets through still has to satisfy your attribute normalization rules, so that a throttled-then-retried response is reconciled by parcel_id rather than written as a duplicate.
• A persistent quota registry. Observed limits must survive process restarts. A registry held only in memory re-learns every jurisdiction’s ceiling from scratch on each deploy — which means re-triggering the throttle it just learned about.

With those in place, the limiter has a stable contract to enforce: pace outbound traffic per jurisdiction, learn each endpoint’s real ceiling from its responses, and degrade to a secondary channel rather than failing the run.

Architecture: registry, gate, and isolation jump to heading

The design has three cooperating parts, each solving a distinct failure mode.

The quota registry maps every jurisdictional endpoint to its observed limit, request window, and compliance tier. Municipal APIs advertise limits through HTTP headers — X-RateLimit-Limit, X-RateLimit-Remaining, Retry-After — but a large share publish nothing and simply start rejecting traffic at an undocumented threshold. The registry therefore starts every unknown endpoint with a deliberately conservative default and self-tunes upward or downward from what each response reveals. This learning loop is the single most important behaviour: the system converges on each server’s real tolerance instead of guessing once and hoping.

The gate is a sliding-window token bucket that runs before a request leaves the application layer. A fixed-window counter is simpler, but it allows a burst at a window boundary — 30 requests at 11:59:59 and 30 more at 12:00:00 reads as two compliant windows yet lands 60 requests on the server in two seconds, which is exactly what trips municipal throttles. A sliding window prunes timestamps older than the window on every check, so the rate is smooth across boundaries.

Per-jurisdiction isolation keeps one failing municipality from poisoning the rest. Each base URL carries its own bucket and its own circuit breaker, so a county that goes down for maintenance opens its breaker and routes to fallback while every other jurisdiction keeps flowing. Without isolation, a single slow server backs up a shared queue and starves the whole refresh.

The flow is: a request consults the registry for the endpoint’s current limit and window, asks the gate for a slot, and is either sent immediately or parked until a slot frees. Every response feeds its headers back into the registry. Repeated rejections trip the breaker, which diverts traffic to a cached or scraped source until a cooldown elapses.

Production implementation jump to heading

The two components below are the load-bearing pieces: a thread-safe sliding-window limiter that self-tunes from response headers, and an async client that consults it, honours Retry-After, and isolates failing jurisdictions behind a circuit breaker. Both are runnable against an httpx.AsyncClient.

import asyncio
import logging
import random
import threading
import time
from collections import deque
from typing import Deque, Dict, Optional

import httpx

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s | %(levelname)s | %(message)s",
)
logger = logging.getLogger(__name__)


class MunicipalRateLimiter:
    """Thread-safe sliding-window limiter that self-tunes from response headers.

    One window per endpoint so jurisdictions never share a budget. Limits start
    conservative and converge on each server's real ceiling as headers arrive.
    """

    def __init__(self, default_limit: int = 30, default_window: float = 60.0):
        self._windows: Dict[str, Deque[float]] = {}
        self._limits: Dict[str, int] = {}
        self._window_sec: Dict[str, float] = {}
        self._lock = threading.Lock()
        self._default_limit = default_limit
        self._default_window = default_window

    def _prune(self, endpoint: str, window_sec: float) -> None:
        cutoff = time.time() - window_sec
        w = self._windows[endpoint]
        while w and w[0] <= cutoff:   # deque is time-ordered; pop from the left
            w.popleft()

    def acquire(self, endpoint: str) -> bool:
        """Try to claim a slot. Returns False when the endpoint is saturated."""
        with self._lock:
            limit = self._limits.get(endpoint, self._default_limit)
            window = self._window_sec.get(endpoint, self._default_window)
            self._windows.setdefault(endpoint, deque())
            self._prune(endpoint, window)
            if len(self._windows[endpoint]) >= limit:
                return False
            self._windows[endpoint].append(time.time())
            return True

    def update_from_headers(self, endpoint: str, headers: dict) -> None:
        """Self-tune the registry from observed municipal response headers."""
        with self._lock:
            limit = headers.get("X-RateLimit-Limit")
            window = headers.get("X-RateLimit-Window")
            if limit is not None:
                try:
                    self._limits[endpoint] = max(1, int(limit))
                except ValueError:
                    pass
            if window is not None:
                try:
                    self._window_sec[endpoint] = max(1.0, float(window))
                except ValueError:
                    pass


class CircuitBreaker:
    """Per-jurisdiction breaker: trips after repeated 5xx, cools down, recovers."""

    def __init__(self, threshold: int = 5, cooldown: float = 300.0):
        self._failures = 0
        self._threshold = threshold
        self._cooldown = cooldown
        self._open_until = 0.0

    @property
    def is_open(self) -> bool:
        return time.time() < self._open_until

    def record_success(self) -> None:
        self._failures = 0

    def record_failure(self) -> None:
        self._failures += 1
        if self._failures >= self._threshold:
            self._open_until = time.time() + self._cooldown
            logger.warning("Circuit opened for %.0fs after %d failures",
                           self._cooldown, self._failures)


class ResilientMunicipalClient:
    """Rate-aware, retry-aware async client for a single municipal base URL."""

    RETRYABLE = {429, 502, 503, 504}

    def __init__(
        self,
        base_url: str,
        limiter: MunicipalRateLimiter,
        max_retries: int = 5,
        timeout: float = 30.0,
    ):
        self.base_url = base_url.rstrip("/")
        self.limiter = limiter
        self.max_retries = max_retries
        self.breaker = CircuitBreaker()
        self._client = httpx.AsyncClient(base_url=self.base_url, timeout=timeout)

    async def _wait_for_slot(self, endpoint: str) -> None:
        """Block on the token bucket with jitter so partitions never sync up."""
        while not self.limiter.acquire(endpoint):
            await asyncio.sleep(random.uniform(0.25, 1.5))

    async def _backoff(self, attempt: int, retry_after: Optional[str]) -> None:
        if retry_after is not None:
            try:                                   # honour the server's cooldown
                await asyncio.sleep(float(retry_after) + random.uniform(0.1, 0.5))
                return
            except ValueError:
                pass
        await asyncio.sleep(min(30.0, 2 ** attempt) + random.uniform(0.0, 0.5))

    async def fetch(self, endpoint: str, params: Optional[dict] = None) -> Optional[dict]:
        """Fetch one endpoint, pacing against the limiter and retrying transients."""
        if self.breaker.is_open:
            raise RuntimeError(f"Circuit open for {self.base_url}; route to fallback")

        for attempt in range(self.max_retries):
            await self._wait_for_slot(endpoint)
            try:
                resp = await self._client.get(endpoint, params=params)
            except (httpx.TimeoutException, httpx.TransportError) as exc:
                self.breaker.record_failure()
                if attempt < self.max_retries - 1:
                    await self._backoff(attempt, None)
                    continue
                logger.error("Transport failure on %s: %s", endpoint, exc)
                return None

            # Always learn from the response, success or not.
            self.limiter.update_from_headers(endpoint, dict(resp.headers))

            if resp.status_code in self.RETRYABLE and attempt < self.max_retries - 1:
                if resp.status_code >= 500:
                    self.breaker.record_failure()
                await self._backoff(attempt, resp.headers.get("Retry-After"))
                continue

            if resp.is_success:
                self.breaker.record_success()
                return resp.json()

            logger.warning("Non-retryable %d on %s", resp.status_code, endpoint)
            return None

        logger.error("Exhausted %d attempts on %s", self.max_retries, endpoint)
        return None

    async def aclose(self) -> None:
        await self._client.aclose()

Three choices are deliberate. The limiter update_from_headers runs on every response, including rejections, so a 429 that ships an X-RateLimit-Limit immediately corrects the registry rather than waiting for a success. fetch honours Retry-After before computing its own backoff, because many municipal gateways return an explicit cooldown that is more accurate than any formula. And only 5xx failures feed the breaker — a 429 is the server pacing you correctly, not failing, so it should slow you down without tripping a jurisdiction offline.

Spatial query optimization jump to heading

The cheapest request is the one you never make, and the second cheapest is the small one. Because municipal limits are frequently tied to payload size, geometry complexity, or query cost rather than a flat request count, shrinking each request stretches the same quota across far more parcels:

1. Bounding-box pre-filtering. Never request a full jurisdiction. Pass a bbox or geometry intersection parameter scoped to active development corridors so the server returns and bills for only the features you need.
2. Explicit field selection. WFS and ArcGIS REST endpoints return dozens of administrative attributes by default. Request only outFields/fields that matter — zoning code, parcel ID, amendment date, status flag — which cuts payload size and the per-row processing the server charges against your quota.
3. Geometry deferral. For tabular tracking, append f=json&returnGeometry=false to skip coordinate serialization entirely. Re-fetch geometry only when a parcel transitions to a target zoning state, so coordinates are paid for once, on change, not on every poll.
4. Cursor-based pagination. Where supported, prefer token cursors over offset/limit. Offset pagination forces the municipal database to re-scan its index on every page, inflating query cost and triggering aggressive throttling on deep pages.

These choices align with GIS export sync workflows: heavy spatial payloads are materialised only when a downstream consumer actually needs the geometry, which keeps the polling path light and the quota intact.

Edge cases and gotchas jump to heading

Municipal rate limiting fails in specific, recurring ways. Plan for these before production:

• Undocumented and silently changing limits. A server with no rate headers may still cap at an internal threshold that shifts during peak planning hours. Never hardcode a ceiling; let the registry learn it, and keep the default conservative (≤15 req/min for legacy servers) so the first request never trips the cap.
• Retry-After in two formats. RFC-compliant servers may return Retry-After as either a delay in seconds (120) or an HTTP date (Wed, 21 Oct 2026 07:28:00 GMT). Parse both; a float() on a date string throws and silently falls through to short backoff, re-tripping the throttle.
• Per-jurisdiction request fan-out. A single global limiter still aimed at one county is no protection if every partition targets that county. Keep one bucket per base URL so partitioning by zoning district does not multiply requests against a single server.
• Shared upstream gateways. Several “different” municipal subdomains sometimes sit behind one vendor gateway (e.g. a shared ArcGIS Online org) with a combined quota. When you see correlated 429s across endpoints, group them under one logical bucket.
• Clock skew on sliding windows. The window prunes on local wall-clock time. A container with drifting time can prune too early or too late; pin to a monotonic source or sync NTP so the window stays accurate.
• Breaker flapping. A breaker that reopens the instant one request after cooldown fails will oscillate. Reset the failure count only after a successful request, and consider a half-open probe before fully closing — this is where error handling & retry logic belongs, scoped per stage rather than bolted on globally.

Throttle response reference jump to heading

Different status codes mean different things on a municipal endpoint, and treating them uniformly either wastes quota or masks real problems.

Integration points jump to heading

Rate-limit management is the contract every fetch upstream depends on. The concurrency ceiling it publishes is what bounds async batch processing — the batch engine sizes its semaphore from this policy rather than guessing, so the two gates (in-flight count and request rate) stay consistent. When the breaker opens after a sustained 429/503 run, the request does not fail the pipeline; it hands off to fallback routing logic, which fills the gap from a secondary feed or last-known-good snapshot. For jurisdictions with no OGC API at all, the secondary channel is the PDF & HTML scraping pipelines, so a quota exhaustion on the structured endpoint degrades to parsing planning-commission agendas rather than going dark. The contract in every direction is the same normalized record shape, so the limiter can be tuned, replayed, or swapped without downstream changes.

Fallback and data-continuity protocols jump to heading

When a quota is exhausted or a server goes down, the pipeline must degrade gracefully rather than halt. A tiered strategy preserves continuity:

• Local spatial cache. Keep a lightweight SQLite/SpatiaLite cache of recently fetched parcels. Serve reads from the cache during a throttle window and queue refreshes for off-peak hours.
• Request queue. A Redis-backed queue serialises requests when the bucket is saturated; workers drain it at a steady, compliant pace instead of retrying in a tight loop.
• Document-level fallback. When structured APIs are unavailable, route zoning-amendment lookups to document scraping, parse statuses from agendas and minutes, and reconcile against cached spatial records.
• Checkpointed pause. If consecutive rejections exceed the breaker threshold, pause the partition, persist checkpoint state, flush in-memory buffers, and resume only after a time-based or manual reset.

Compliance and audit artifacts jump to heading

For PropTech underwriting and regulatory review, a rate-limited run is provable only if it records how it behaved against each jurisdiction. Every run should emit:

• A throttle ledger: every 429/503/Retry-After event with endpoint, timestamp, observed limit, and the backoff applied — the evidence that ingestion stayed within each municipality’s terms of service.
• A registry snapshot: the learned limit and window per endpoint at run start and end, so quota trends are visible over time and a sudden tightening is detectable.
• Breaker transitions: every open/close with the failure count and cooldown, linking any data gap to a specific jurisdiction outage rather than a silent miss.
• Per-jurisdiction request counts: issued, throttled, queued, and served-from-cache, reconcilable against what reached the database.

These artifacts pair with downstream schema validation & data quality checks to form a defensible chain from the first outbound request to an underwriting-grade dataset.

FAQ jump to heading

Related jump to heading

• Async batch processing — consumes the concurrency budget this limiter publishes
• Error handling & retry logic — per-stage failure boundaries and recovery semantics
• PDF & HTML scraping pipelines — the document fallback when a structured API is throttled
• GIS export sync workflows — keeps the polling path light by materialising geometry late
• Fallback routing logic — where an open circuit breaker hands off

Up: Automated Feed Ingestion & GIS Data Parsing