aiohttp vs httpx for municipal API polling

You are polling several hundred municipal endpoints on a schedule — ArcGIS REST services, GeoServer WFS, Socrata datasets, and a sprawling assortment of bespoke county portals — every one of them behind a different rate limit, a different timeout personality, and a different idea of what a well-formed response is. The async HTTP client you pick shapes how gracefully that whole operation degrades: whether you can hold thousands of connections open without exhausting file descriptors, whether a single hung host stalls a batch, and how cleanly you can bolt on retries and municipal API rate limit management without rewriting every call site. This guide frames the choice between the two dominant libraries — aiohttp and httpx — as a decision rather than a benchmark, because for the polling workloads inside an automated feed ingestion and GIS data parsing pipeline the differentiators are ergonomics and control, not raw microseconds.

Diagnosis: symptoms that push you to switch jump to heading

Most teams do not choose a client deliberately; they inherit one and hit a wall. Recognize which wall you are at, because it points directly at the better fit.

• You need HTTP/2 and your client cannot speak it. A modern gateway fronting a county’s ArcGIS server multiplexes many requests over one HTTP/2 connection. On aiohttp, which is HTTP/1.1-only, each concurrent request to that host needs its own TCP+TLS connection, and you burn connections and handshake latency that a single HTTP/2 stream would absorb. The symptom is connection-pool exhaustion and TLS overhead against a host you know supports multiplexing.
• Your retry and rate-limit logic is copy-pasted into every call. When backoff, Retry-After handling, and header logging live inside each await client.get(...), adding a circuit breaker means editing dozens of sites. The symptom is that cross-cutting policy cannot be expressed in one place.
• The same code must run sync in a Celery task and async in the poller. Reusing one request-building and response-parsing layer across a synchronous ingestion job and an asynchronous poller is painful when the client has no sync twin. The symptom is two parallel, drifting implementations.
• One slow host stalls the whole batch. A single county server that accepts your connection then never sends bytes will hang a naive request forever. The symptom is a batch that never completes because one endpoint has no granular timeout. Both clients solve this — but how you configure it differs, and getting it wrong is common.

The decision flow below maps the questions that actually separate the two, and the sections after it turn each into a concrete recommendation.

The decision matrix jump to heading

The table weighs the dimensions that matter for polling rate-limited municipal endpoints. Read it as “which client makes this concern easier,” not as a scoreboard.

Two rows carry most of the decision. HTTP/2 is a hard capability gap: if your highest-volume hosts sit behind multiplexing gateways, only httpx exploits it. Transport hooks are an ergonomics gap: httpx lets you express retries, Retry-After handling, and logging once in an event hook or custom transport, whereas on aiohttp that policy lives at each call site unless you build a wrapper.

When to pick which jump to heading

• Pick httpx when policy and protocol flexibility dominate. Hundreds of heterogeneous portals, several behind HTTP/2 gateways, with retries and rate-limit accounting you want centralized, are exactly its sweet spot. The transport layer and event hooks give you one place to attach a circuit breaker and header-aware backoff, and MockTransport makes the whole polling path unit-testable without a live server.
• Pick aiohttp when you are fanning out to thousands of HTTP/1.1 hosts and every microsecond of loop overhead counts. Its connector model is mature and its overhead per request is minimal, so a very large, homogeneous batch of simple GETs runs lean. It is also the natural choice if the rest of your service already lives in the aiohttp web-server ecosystem.
• Pick httpx when the same code must run sync and async. A Celery worker doing synchronous backfill and an async poller can share one request-building and response-parsing module because the sync Client and AsyncClient present the same API — no second implementation to keep in sync.
• When neither dimension bites, let team familiarity decide. For a few hundred well-behaved HTTP/1.1 endpoints with no HTTP/2 and modest policy needs, both clients are more than capable; the maintainable choice is the one your team already knows.

Equivalent polling snippets jump to heading

The mechanics are close enough that porting is mostly mechanical. Both examples poll one endpoint with split connect/read timeouts and bounded pooling — the primitives you build rate limiting and retries on top of.

aiohttp jump to heading

import asyncio

import aiohttp


async def poll_aiohttp(urls: list[str], per_host: int = 4) -> dict[str, int]:
    # split timeouts: cap connect and single-read so one hung host can't stall the batch
    timeout = aiohttp.ClientTimeout(total=30, connect=5, sock_read=10)
    connector = aiohttp.TCPConnector(limit=100, limit_per_host=per_host)
    results: dict[str, int] = {}

    async with aiohttp.ClientSession(timeout=timeout, connector=connector) as session:

        async def fetch(url: str) -> None:
            try:
                async with session.get(url) as resp:
                    await resp.read()  # drain so the connection returns to the pool
                    results[url] = resp.status
            except (aiohttp.ClientError, asyncio.TimeoutError) as exc:
                results[url] = -1  # sentinel: hand off to retry/dead-letter logic
                print(f"aiohttp failed {url}: {exc!r}")

        await asyncio.gather(*(fetch(u) for u in urls))

    return results

httpx jump to heading

import asyncio

import httpx


async def poll_httpx(urls: list[str], per_host: int = 4) -> dict[str, int]:
    # four independent timeout phases; http2 pays off against multiplexing gateways
    timeout = httpx.Timeout(30.0, connect=5.0, read=10.0)
    limits = httpx.Limits(max_connections=100, max_keepalive_connections=per_host)
    results: dict[str, int] = {}

    async with httpx.AsyncClient(
        timeout=timeout, limits=limits, http2=True
    ) as client:

        async def fetch(url: str) -> None:
            try:
                resp = await client.get(url)
                results[url] = resp.status_code
            except (httpx.HTTPError, asyncio.TimeoutError) as exc:
                results[url] = -1  # sentinel: hand off to retry/dead-letter logic
                print(f"httpx failed {url}: {exc!r}")

        await asyncio.gather(*(fetch(u) for u in urls))

    return results

The httpx version adds a real transport-level retry on connection errors with one line, which is the ergonomic edge that matters when policy must be centralized:

async def build_retrying_client() -> httpx.AsyncClient:
    # retries connection failures at the transport layer, not 5xx responses
    transport = httpx.AsyncHTTPTransport(retries=3)
    # every request through this client now retries transient connect failures
    return httpx.AsyncClient(transport=transport, http2=True)

Verification & testing jump to heading

Confirm the client you chose actually behaves under the failure modes municipal endpoints throw at you.

• Read timeouts fire independently of the total. Point the poller at an endpoint that accepts the connection then stalls; assert the request fails at sock_read / read, not after the full total, so one dead host cannot pin a worker.
• Pooling is bounded. Fan out well past limit_per_host / max_keepalive_connections concurrent requests to one host and confirm connection count plateaus at the cap instead of climbing toward file-descriptor exhaustion.
• HTTP/2 is really negotiated. On httpx, assert resp.http_version == "HTTP/2" against a known multiplexing host; if it reports HTTP/1.1, the h2 extra is missing and you gained nothing.
• The sentinel routes downstream. Force a failure and assert the -1 (or your typed error) reaches the retry and dead-letter path from error handling and retry logic rather than being swallowed.
• Tests run without a live server. With httpx, use MockTransport to script 429s, timeouts, and 500s and assert your rate-limit handling reacts correctly; on aiohttp, stand up aiohttp.test_utils fixtures for the same coverage.

Failure recovery jump to heading

Whichever client you standardize on, the recovery posture is the same; only the hook points differ.

• Honor Retry-After in one place. Centralize 429 handling so a Retry-After header pauses the offending host, not the whole batch — on httpx this lives naturally in an event hook or custom transport; on aiohttp it goes in your call wrapper. This is the core of municipal API rate limit management.
• Add a circuit breaker per host. Neither client stops hammering a failing endpoint on its own; wrap either with the per-host breaker from implementing circuit breakers for flaky municipal endpoints so a flap cannot become an IP ban.
• Cap concurrency separately from pooling. Pool limits control open sockets, not in-flight work; gate the fan-out with a Semaphore sized to the slowest host as covered in async batch processing, so a burst of small responses does not overrun a fragile portal.
• Do not migrate for microseconds. If your only symptom is a wish for slightly higher throughput, switching clients rarely repays the rewrite. Migrate when you need HTTP/2, sync+async parity, or transport hooks — capability gaps, not benchmark deltas.

Frequently asked questions jump to heading

Related jump to heading

• Parent topic: Municipal API Rate Limit Management
• Section overview: Automated Feed Ingestion & GIS Data Parsing
• Implementing circuit breakers for flaky municipal endpoints — wrap either client with a per-host breaker
• Error Handling & Retry Logic — where the failure sentinel is routed
• Async Batch Processing — bounding concurrency independently of the connection pool