Handling Retry-After headers that lie

The spec is simple: on a 429, wait the number of seconds in Retry-After and try again. Municipal portals make it complicated. One returns Retry-After: 0, so a compliant client retries immediately and is rate-limited again in a tight loop. One returns 3600 on every 429 regardless of the actual window, so a compliant client sleeps an hour for a limit that resets in twenty seconds. One returns an HTTP date in the past. One returns the header on a 200. Honouring a lying header is worse than ignoring it, and ignoring it is worse than honouring a truthful one — so the client has to do both, carefully. This guide sanitises the header and learns the real limit by observation, extending municipal API rate limit management.

Diagnosis: characterise the header before trusting it

Log every Retry-After you receive alongside how long it actually took for requests to succeed again. The comparison tells you which category a portal is in.

Five kinds of Retry-After behaviour, and what each one needs A five-row by three-column matrix of Retry-After header behaviour. Rows are a truthful header, a header that is always zero, a constant overstated header, an unreliable header, and an absent header. Columns are what the portal sends, what happens if the client obeys it literally, and the correct response. Only the truthful row can be obeyed as sent; obeying a zero produces a request loop and obeying a constant overstated value stalls the run for an hour. Five kinds of Retry-After behaviour, and what each one needs what the portal sends if obeyed literally correct response Truthful close to real recovery correct backoff honour it, clamped Always zero Retry-After: 0 a hot request loop ignore; use own backoff Constant overstated 3600 on a 20s window the run stalls an hour clamp to a ceiling Unreliable uncorrelated values sometimes too fast max(header, backoff) Absent no header at all nothing to obey backoff, and learn safe needs a hint actively harmful
Only the first row can be obeyed as sent. The two rows below it are the ones that cause real damage in opposite directions — a hot request loop and an hour-long stall — and both are produced by a client that follows the spec exactly.
from dataclasses import dataclass


@dataclass
class RetryAfterObservation:
    host: str
    header_raw: str | None
    parsed_seconds: float | None
    actual_recovery_seconds: float      # measured: 429 until the next 2xx
    status: int


def characterise(observations, host) -> str:
    """Which kind of liar is this host?"""
    obs = [o for o in observations if o.host == host and o.status == 429]
    if not obs:
        return "no 429s observed"
    with_header = [o for o in obs if o.parsed_seconds is not None]
    if not with_header:
        return "absent"                    # no header at all: infer everything

    ratios = [o.actual_recovery_seconds / max(o.parsed_seconds, 0.001)
              for o in with_header]
    median = sorted(ratios)[len(ratios) // 2]
    distinct = {o.parsed_seconds for o in with_header}

    if all(v == 0 for v in distinct):
        return "always_zero"               # honouring it is a hot loop
    if len(distinct) == 1 and median < 0.1:
        return "constant_overstated"       # e.g. always 3600 for a 20-second window
    if 0.5 <= median <= 2.0:
        return "truthful"
    return "unreliable"

The five categories each need different handling, and the important insight is that only one of them means “do what the header says.”

Category What the portal sends Correct response
truthful a value close to the real recovery time honour it, clamped
always_zero 0 on every 429 ignore it; use your own backoff
constant_overstated the same large value regardless clamp to a learned ceiling
unreliable varies, uncorrelated with reality treat as a hint; take the max of it and your backoff
absent no header pure backoff, and learn the limit

Step-by-step implementation

1. Parse both forms, and refuse nonsense 

import email.utils
from datetime import datetime, timezone

MIN_SLEEP = 1.0          # a compliant 0 is a hot loop; never sleep less than this
MAX_SLEEP = 300.0        # a 3600 on a 20-second window stalls the whole run


def parse_retry_after(value: str | None, *, now=None) -> float | None:
    """Returns seconds, or None if the header is absent or unparseable.

    Both forms are legal: delta-seconds, and an HTTP-date. A date in the past
    parses fine and means zero, which is one of the ways the header lies.
    """
    if value is None:
        return None
    value = value.strip()
    if not value:
        return None

    if value.isdigit():
        return float(value)

    try:
        when = email.utils.parsedate_to_datetime(value)
    except (TypeError, ValueError):
        return None
    if when is None:
        return None
    if when.tzinfo is None:
        when = when.replace(tzinfo=timezone.utc)
    now = now or datetime.now(timezone.utc)
    return max(0.0, (when - now).total_seconds())


def sanitise(seconds: float | None, backoff: float, category: str) -> tuple[float, str]:
    """Combine the header with our own backoff according to what this host does.
    Returns (sleep_seconds, reason) — the reason goes in the log so a decision can
    be explained later."""
    if category == "truthful" and seconds is not None:
        return max(MIN_SLEEP, min(seconds, MAX_SLEEP)), "header (truthful, clamped)"
    if category == "always_zero":
        return backoff, "header ignored (always zero)"
    if category == "constant_overstated" and seconds is not None:
        return max(MIN_SLEEP, min(backoff * 2, MAX_SLEEP)), "header clamped (overstated)"
    if seconds is None:
        return backoff, "no header"
    # Unreliable: the header is a hint, and the safe reading is the larger value.
    return max(MIN_SLEEP, min(max(seconds, backoff), MAX_SLEEP)), "max(header, backoff)"

MIN_SLEEP is the single most important line. A portal sending Retry-After: 0 and a client that honours it produce a request loop bounded only by network latency, which is indistinguishable from an attack and gets your IP blocked.

What the clamp does to a header that lies Line chart comparing the sleep a client would take against the header value it received, from 0 to 600 seconds. The unclamped line follows the header exactly up to 600 seconds. The clamped line starts at a one-second floor where the header says zero, tracks the header exactly from 1 to 300 seconds, then flattens at the 300-second ceiling. A header of 3600 clamps to the same ceiling. What the clamp does to a header that lies 0 200 400 600 800 0 s 1 s 5 s 30 s 120 s 300 s 450 s 600 s Retry-After value received Seconds actually slept sleep taken, unclamped sleep taken, clamped
The clamp only changes behaviour at the two ends. Between roughly 1 and 300 seconds the client does exactly what the header asks; outside that band it refuses, because a zero is a request loop and an hour is a scheduling decision rather than a sleep.

2. Learn the real limit from the 429s themselves 

from collections import deque


class LearnedLimit:
    """Infer a host's actual rate limit by watching when 429s start. More reliable
    than documentation, which municipal portals rarely have and rarely update."""

    def __init__(self, window_seconds=60, history=200):
        self.window = window_seconds
        self.successes = deque(maxlen=history)     # timestamps of 2xx
        self.rejections = deque(maxlen=history)    # timestamps of 429

    def record(self, ts, ok: bool):
        (self.successes if ok else self.rejections).append(ts)

    def observed_ceiling(self, now) -> int | None:
        """The highest number of successful requests seen in any window that did NOT
        end in a 429 — a lower bound on the real limit."""
        if not self.rejections:
            return None
        cutoff = now - self.window
        recent_ok = [t for t in self.successes if t >= cutoff]
        first_429 = min((t for t in self.rejections if t >= cutoff), default=None)
        if first_429 is None:
            return None
        return sum(1 for t in recent_ok if t < first_429)

    def suggested_rate(self, now) -> float | None:
        ceiling = self.observed_ceiling(now)
        if ceiling is None:
            return None
        # Stay at 80% of the observed ceiling: enough headroom that a burst does not
        # immediately re-trip the limit.
        return 0.8 * ceiling / self.window
Documented limit against the limit actually enforced Grouped bar chart comparing the documented rate limit with the observed ceiling for four county portals, in requests per minute. The first portal documents 60 and enforces 40. The second documents nothing and enforces 100. The third documents 120 and enforces 118. The fourth documents 30 and enforces 12. Only the third portal matches its own documentation. Documented limit against the limit actually enforced 0 50 100 150 60 40 King 0 100 Pierce 120 118 Snohomish 30 12 Kitsap Requests per minute documented limit observed ceiling
One portal of the four matches its documentation. Two enforce well below the published number and one publishes nothing at all — which is why the ceiling is measured from observed 429s rather than read from a page that was last edited in 2019.

3. Wire it into the retry loop 

import logging
import random

log = logging.getLogger(__name__)


async def fetch_with_limit(session, url, host_state, *, attempts=5):
    for attempt in range(1, attempts + 1):
        resp = await session.get(url)
        host_state.limit.record(now(), ok=resp.status < 400)

        if resp.status != 429:
            return resp

        backoff = random.uniform(0, min(60.0, 0.5 * 2 ** (attempt - 1)))   # full jitter
        raw = resp.headers.get("Retry-After")
        seconds = parse_retry_after(raw)
        sleep_for, why = sanitise(seconds, backoff, host_state.category)

        log.warning("%s 429 (attempt %d): Retry-After=%r → sleeping %.1fs [%s]",
                    host, attempt, raw, sleep_for, why)
        host_state.observe(RetryAfterObservation(host, raw, seconds, None, 429))
        await asyncio.sleep(sleep_for)

    raise RateLimited(f"{host}: still 429 after {attempts} attempts")

4. Stay a good citizen when the server is wrong

Ignoring a header is a decision with an ethical dimension, so it needs guardrails: never retry faster than MIN_SLEEP, keep the per-host concurrency cap in place regardless of what the header says, cap total attempts, and record every ignored header with the reason. If a portal complains, the log shows a client that backed off on every 429 and only declined to follow a header that said zero.

Verification & testing 

@pytest.mark.parametrize("raw,expected", [
    ("120", 120.0),
    ("0", 0.0),
    ("", None),
    (None, None),
    ("banana", None),
    ("Wed, 21 Oct 2026 07:28:00 GMT", 0.0),          # a date in the past
])
def test_parsing(raw, expected):
    got = parse_retry_after(raw, now=datetime(2026, 11, 1, tzinfo=timezone.utc))
    assert got == expected


def test_zero_never_becomes_a_hot_loop():
    sleep_for, why = sanitise(0.0, backoff=4.0, category="always_zero")
    assert sleep_for >= MIN_SLEEP
    assert "ignored" in why


def test_overstated_header_is_clamped():
    sleep_for, _ = sanitise(3600.0, backoff=4.0, category="constant_overstated")
    assert sleep_for <= MAX_SLEEP and sleep_for < 3600


def test_truthful_header_is_honoured():
    sleep_for, why = sanitise(45.0, backoff=2.0, category="truthful")
    assert sleep_for == 45.0 and "truthful" in why


def test_unreliable_takes_the_larger_value():
    assert sanitise(3.0, backoff=12.0, category="unreliable")[0] == 12.0
    assert sanitise(30.0, backoff=4.0, category="unreliable")[0] == 30.0


def test_learned_ceiling_tracks_observed_429s():
    lim = LearnedLimit(window_seconds=60)
    t0 = 1000.0
    for i in range(40):
        lim.record(t0 + i, ok=True)
    lim.record(t0 + 41, ok=False)                    # the 41st request trips it
    assert lim.observed_ceiling(t0 + 45) == 40
    assert lim.suggested_rate(t0 + 45) == pytest.approx(0.8 * 40 / 60)

Failure recovery

An IP blocked after honouring Retry-After: 0. Contact the portal, then fix MIN_SLEEP before re-enabling the source. Bring the source back at a deliberately low learned rate — the block came from request volume, and the operator will be watching.

A run stalling for an hour on a constant overstated header. Clamp to MAX_SLEEP and re-characterise the host. The measured recovery time is the number to trust; the header is a claim.

A host whose category has changed. Portals get upgraded, and a previously truthful header can become a constant. Re-characterise on a rolling window rather than pinning the category once, and alert when the median ratio moves by more than a factor of two.

Frequently asked questions

Is it acceptable to ignore Retry-After?

When the header is provably wrong, yes — with guardrails. A header saying zero cannot be honoured without producing a request loop, which is worse for the server than backing off on your own schedule. What is not acceptable is ignoring it and retrying aggressively: the floor, the per-host concurrency cap and the attempt limit all stay in force, and every ignored header is logged with a reason.

Why learn the limit instead of reading the documentation?

Because municipal portals rarely document a limit and, when they do, it is frequently out of date or describes a different tier than the one you are on. Observing when 429s begin measures the limit that is actually enforced today. Staying at 80% of the observed ceiling leaves enough headroom that an ordinary burst does not immediately re-trip it.

Should the header cap be as low as five minutes?

For an ingestion run, yes. A genuine limit that needs longer than five minutes means the source cannot be consumed in this run at all, which is a scheduling decision rather than a sleep — hold the jurisdiction and let the fallback tier serve, instead of blocking a worker for an hour.

What if a portal sends Retry-After on a 200?

Ignore it. The header is only meaningful with a 429 or a 503; on a success it is either a misconfiguration or a proxy artefact. Acting on it would insert sleeps into a healthy path, which is the sort of thing that makes a run mysteriously slow with no failures to point at.