Staggering per-jurisdiction schedules to avoid a thundering herd

Every source in your registry has the same schedule string, 0 2 * * *, because that is what you wrote when there were two of them and there are now thirty-one. At 02:00:00 all thirty-one wake, open connections, and hit whatever they share — a state-hosted ArcGIS instance serving nine of the counties, one geocoder, one database connection pool, one egress NAT. The pipeline is not overloaded on average; it is overloaded for ninety seconds and idle for the rest of the day. Worse, everything that fails in that window retries on the same backoff schedule, so the second spike arrives together too. This guide spreads the load deterministically. It is the load-shaping half of scheduling & orchestration for municipal feed runs, and it pairs with the per-host bulkheads in municipal API rate limit management.

Diagnosis: proving it is a herd and not a capacity problem

The distinguishing symptom is that failures cluster in time rather than by source. Three checks separate the two causes.

Plot request starts by second-of-minute. A herd shows a spike at :00 of the scheduled minute and nothing for the rest of the hour. A genuine capacity problem is flat and always slightly over.

Where 31 sources land in a 50-minute window Grouped bar chart of how 31 sources distribute across ten five-minute buckets in a fifty-minute window, comparing the unstaggered schedule where all 31 start in the first bucket against the hash-derived offsets, which spread them across every bucket with between one and six sources each. The offsets are computed at build time with the same SHA-256 function the guide uses, so the distribution shown is the one the code actually produces. Where 31 sources land in a 50-minute window 0 10 20 30 40 31 2 0–5 0 1 5–10 0 3 10–15 0 4 15–20 0 5 20–25 0 3 25–30 0 8 30–35 0 1 35–40 0 1 40–45 0 3 45–50 Sources starting in the bucket all at 02:00 (unstaggered) hash-derived offsets Minutes into the window. Offsets from sha256(source_id), as in the code above.
Computed at build time with the same hash function as the code above: 31 sources spread across every five-minute bucket, one to six per bucket. The unstaggered column is the herd — 31 simultaneous starts against whatever those sources share.

Group errors by upstream host, then by minute. If nine counties served by one state ArcGIS instance all fail within the same ninety seconds while twenty-two others succeed, the shared upstream is saturated — not your workers.

Check whether the retries also cluster. This is the clincher. Failures at 02:00:15 that all retry at 02:00:16, then 02:00:18, then 02:00:22 are synchronised by a shared un-jittered backoff, and each retry wave is itself a smaller herd.

-- Requests per second within the scheduled minute, by upstream host.
SELECT upstream_host,
       date_trunc('second', started_at) AS at,
       count(*)                          AS requests,
       count(*) FILTER (WHERE status >= 500 OR status = 429) AS failures
  FROM fetch_log
 WHERE started_at >= now() - interval '1 day'
   AND extract(minute FROM started_at) = 0
   AND extract(hour   FROM started_at) = 2
 GROUP BY 1, 2
 ORDER BY requests DESC
 LIMIT 20;

If the top row shows twenty-plus requests in one second against one host, the fix is scheduling, not scaling.

Step-by-step implementation

1. Derive a stable offset from the source id

The offset must be deterministic — the same source lands in the same slot every day, so its behaviour is comparable across runs — and it must be stable across process restarts, which rules out random.

import hashlib
from datetime import timedelta


def source_offset(source_id: str, window_minutes: int = 50) -> timedelta:
    """A stable, uniformly distributed offset inside the window.

    Hashing the source id rather than using an index means adding or removing a
    county does not reshuffle everyone else's slot — which matters because a
    reshuffle invalidates every baseline you have for those sources.
    """
    digest = hashlib.sha256(source_id.encode()).digest()
    total_seconds = window_minutes * 60
    return timedelta(seconds=int.from_bytes(digest[:4], "big") % total_seconds)

Using hash() here would be wrong: Python randomises string hashing per process, so offsets would change on every restart.

2. Spread within a window, not across the whole day

Municipal portals publish overnight and many are noticeably faster before local business hours, so the goal is to spread inside a window rather than to scatter over twenty-four hours.

def next_run_at(source_id, window_start_hour=2, window_minutes=50, now=None):
    now = now or datetime.now(timezone.utc)
    base = now.replace(hour=window_start_hour, minute=0, second=0, microsecond=0)
    at = base + source_offset(source_id, window_minutes)
    return at if at > now else at + timedelta(days=1)

Thirty-one sources spread over fifty minutes averages one start every ninety-seven seconds, which any shared upstream absorbs without noticing.

3. Cap concurrency per upstream host, not per source

Staggering reduces collisions; it does not prevent them, because a slow source still overlaps the next one. The ceiling has to be enforced where the contention actually is.

Gaps between consecutive source starts Lollipop chart of the smallest, first-quartile, median, third-quartile and largest gaps between consecutive source starts under the hash-derived offsets for 31 sources in a fifty-minute window. The values are computed from the same offsets as the histogram. The mean gap is about 97 seconds, and the smallest observed gap is the number worth watching, because it is what determines whether two sources ever collide. Gaps between consecutive source starts 0 100 200 300 400 500 smallest gap 5 s lower quartile 26 s median gap 66 s upper quartile 155 s largest gap 381 s Seconds between consecutive starts Computed from the same hash-derived offsets; watch the minimum as sources are added.
The mean gap is arithmetic — 50 minutes over 31 sources — but the minimum is the number that matters, because that is the pair most likely to overlap. Track it as sources are added; when it approaches a typical fetch duration, widen the window rather than lowering the host limit.
from collections import defaultdict

# Nine counties are served by one state-hosted instance. The limit belongs to the
# HOST, so those nine share six permits between them while a county on its own
# infrastructure is unaffected by their behaviour.
HOST_LIMITS = defaultdict(lambda: 8, {
    "gis.example-state.gov": 6,
    "maps.largecounty.gov": 4,
})


class HostGate:
    def __init__(self):
        self._sems = {}

    def semaphore(self, host: str):
        if host not in self._sems:
            self._sems[host] = asyncio.Semaphore(HOST_LIMITS[host])
        return self._sems[host]

    async def fetch(self, session, url, host):
        async with self.semaphore(host):
            return await session.get(url)

4. Jitter the retries too

An un-jittered retry schedule re-synchronises everything that failed together. Full jitter — sleeping a uniform draw from the whole window rather than the window itself — costs half the delay on average and breaks the correlation.

Full jitter spreads a retry wave that a fixed backoff re-synchronises Shaded-band chart over six retry attempts. The band spans zero to the capped exponential window, which doubles from half a second at attempt one to sixteen seconds at attempt six. The mid line is the mean sleep under full jitter, half the window. Under a fixed backoff every client sleeps the window exactly and returns to the wire together at the top edge of the band; under full jitter each client draws uniformly from the band and the wave is spread across it. Full jitter spreads a retry wave that a fixed backoff re-synchronises 0 5 10 15 20 1 2 3 4 5 6 Retry attempt Sleep window (seconds) legal sleep window mean sleep under full jitter Fixed backoff = the top edge, for everyone, at the same instant.
A fixed backoff puts every retrying worker back on the wire at the top edge of the band, together — the synchronised second wave that takes a recovering portal down again. Drawing uniformly from the band spreads them and costs half the delay on average.
import random


def retry_delay(attempt: int, base: float = 0.5, cap: float = 60.0) -> float:
    """Full jitter. Sleeping the full window is what produces the synchronised
    second wave that takes a recovering portal straight back down."""
    window = min(cap, base * (2 ** (attempt - 1)))
    return random.uniform(0, window)

This is the one place randomness is correct rather than a liability: the offset must be deterministic so schedules are comparable, and the retry must be random so failures decorrelate.

5. Keep the stagger visible in the registry

Store the computed offset alongside each source so an operator can see the plan without running the hash function in their head, and so an unexpected collision is diagnosable from the table rather than from logs.

Verification & testing 

def test_offsets_are_stable_and_spread():
    ids = [f"county-{i:02d}" for i in range(31)]
    offs = sorted(source_offset(i).total_seconds() for i in ids)

    assert offs == sorted(source_offset(i).total_seconds() for i in ids)  # deterministic
    gaps = [b - a for a, b in zip(offs, offs[1:])]
    assert min(gaps) > 5, "two sources land within five seconds of each other"
    assert max(gaps) < 600, "a ten-minute hole means the window is under-used"


def test_adding_a_source_does_not_move_the_others():
    before = {i: source_offset(i) for i in ("larimer", "boulder", "weld")}
    _ = source_offset("new-county")
    assert {i: source_offset(i) for i in ("larimer", "boulder", "weld")} == before


def test_retry_delays_decorrelate():
    delays = [retry_delay(3) for _ in range(200)]
    assert len(set(round(d, 3) for d in delays)) > 150     # not a fixed schedule
    assert all(0 <= d <= 2.0 for d in delays)              # inside the attempt-3 window

In production the check to keep is the maximum requests-per-second against any single upstream host, as a daily series. Staggering that works shows a flat line near your per-host limit; staggering that has quietly stopped working shows a daily spike returning as sources are added.

Failure recovery

A herd is happening right now. Lower the per-host semaphore first — it takes effect on the next request, while a schedule change takes effect tomorrow. Then re-derive offsets, and only then consider whether capacity was ever the issue.

Two important sources collide. Do not hand-pick their offsets, or the next person to add a source will collide with your exception. Widen the window instead, or add a salt to the hash input and re-derive for everyone — the offsets are meant to be a function, not a table of decisions.

A portal complains about your traffic. Reduce that host’s limit and lengthen its interval, then record the agreed rate in the registry next to the source. An undocumented informal agreement is one that the next engineer will breach.

The window is full. Thirty-one sources over fifty minutes is comfortable; ninety is not. Widen the window, or split into two windows on different hours for sources whose portals tolerate it — but check the publication times first, since a source polled before it publishes is a wasted slot.

Frequently asked questions

Why hash the source id instead of using its position in a list?

Because a positional offset reshuffles every source whenever one is added or removed, which invalidates every per-source baseline you have — timing, duration, failure rate. A hash of the id gives each source a slot that belongs to it permanently, so adding a county perturbs nobody else’s schedule.

Is staggering enough on its own?

No. Staggering reduces the probability of collision; it does not bound concurrency, because a slow source still overlaps its successor. You need both: offsets to spread the starts, and a per-upstream-host semaphore to bound what happens when they overlap anyway. The host limit is the guarantee; the stagger just means you rarely reach it.

Should the retry delay be deterministic like the offset?

The opposite. The offset must be deterministic so a source’s schedule is stable and comparable; the retry must be random so that everything which failed together stops retrying together. Sleeping the full exponential window synchronises the second wave, which is what turns a brief upstream wobble into a sustained outage.

How wide should the window be?

Wide enough that the mean gap between starts comfortably exceeds a typical fetch duration, and narrow enough to stay inside the hours when portals are fast and publishing has finished. Thirty-one sources over fifty minutes gives a ninety-seven-second mean gap, which is ample for fetches measured in seconds. Track the minimum observed gap rather than trusting the arithmetic.