Handling HTML table drift in county planning portals

Your nightly scraper has pulled a county planning portal’s zoning table for eighteen months on a CSS selector like table#results tbody tr. One morning it returns zero rows. Nothing errored — the request was 200, the page rendered fine in a browser — but overnight the county migrated its portal to a new framework that renders the same zoning data as a stack of <div class="grid-row"> elements instead of a real <table>, and renamed Zoning Code to Zoning District while it was at it. Your selector matched nothing, your job logged success, and a day of parcel updates silently vanished. This guide answers one narrow question: how do you build HTML scraping that survives a portal redesign inside the wider automated zoning change and municipal GIS tracking workflow? The fix is to anchor extraction on stable content signals instead of brittle DOM paths, normalize headers through a synonym map, score each extraction with a structure hash and confidence metric, and alert the moment the layout shifts under you.

Diagnosis: why a working scraper silently returns nothing

A DOM-path selector encodes an assumption about markup structure that the county never promised to keep. Portal vendors reflow HTML on every redesign, and the failures are quiet because an empty result set is indistinguishable from a legitimately empty page. The breakage takes three recognizable forms.

  1. Structural migration. A semantic <table> becomes CSS-grid <div>s, or vice versa. pandas.read_html, which only parses real <table> elements, returns an empty list; a tbody tr selector matches nothing. The scraper reports zero rows and exits clean.
  2. Column rename or reorder. The table survives but Zoning Code becomes Zoning District, or the Parcel and Acreage columns swap positions. Positional indexing (cells[2]) now reads acreage into the district field. Every row parses successfully and every value is wrong — the most dangerous class, because validation on individual fields often still passes.
  3. Wrapper churn. The data is unchanged but the container id changes from results to data-table-v2, or a new ad wrapper shifts the nesting depth. An absolute path like div.content > div:nth-child(3) > table breaks even though the table is still on the page.

Reproduce the failure by asserting on content, not on the request status. Fetch, attempt extraction, and log the structural fingerprint so a change is visible in the logs before it corrupts data:

import requests
from bs4 import BeautifulSoup

resp = requests.get("https://planning.example-county.gov/zoning", timeout=30)
resp.raise_for_status()
soup = BeautifulSoup(resp.text, "lxml")

n_tables = len(soup.find_all("table"))
n_grid_rows = len(soup.select("[class*=grid-row], [class*=row]"))
headers = [th.get_text(strip=True) for th in soup.select("th")]
print(f"tables={n_tables} grid_rows={n_grid_rows} headers={headers}")

A run where tables drops to zero while grid_rows jumps confirms a structural migration; a run where headers changed confirms a rename. Both are invisible to an HTTP-status check, which is why drift-resilient extraction anchors on these signals directly.

Drift-resilient extraction with structure hashing and confidence routing Fetched portal HTML enters an anchor-finding step that locates the data block by header text and parcel-ID regex rather than a DOM path. A structure hash of the header set is compared against a stored baseline; a match proceeds, a mismatch raises a drift alert. Rows then pass through header-map normalization into canonical fields, and each row receives a confidence score. High-confidence rows are written to the target table while low-confidence rows route to a human review queue. hash mismatch match low conf high conf fetch HTML portal page find anchor header · parcel regex structure hash vs baseline header-map normalize synonyms → canonical confidence score per row drift alert page + pause job review queue low-confidence rows target table high-confidence rows
Extraction anchors on content, hashes the structure against a baseline, and routes rows by confidence so drift alerts before it corrupts data.

Step-by-step implementation

Each step removes one brittle assumption. The orchestrator in the final step composes them.

Step 1 — Anchor on content, not DOM paths

Never navigate by nth-child or a fixed id. Instead, locate the data block by the content it must contain: a header cell whose text matches a zoning header, or a run of cells matching a parcel-ID regex. Search both <table> and <div> grids so a structural migration does not blind you.

import re
from bs4 import BeautifulSoup

PARCEL_ID = re.compile(r"\b\d{2,3}-\d{2,4}-\d{2,4}\b")
HEADER_HINTS = {"parcel", "apn", "zoning", "district", "acreage", "lot"}

def find_data_block(soup: BeautifulSoup):
    """Return the element most likely to hold the zoning table.

    Prefer a real <table> whose header row contains zoning hints; otherwise
    fall back to the densest container of parcel-ID matches.
    """
    for table in soup.find_all("table"):
        header_text = " ".join(th.get_text(" ", strip=True).lower()
                                for th in table.select("th"))
        if sum(hint in header_text for hint in HEADER_HINTS) >= 2:
            return ("table", table)

    # No qualifying table: find the container with the most parcel IDs.
    best, best_count = None, 0
    for container in soup.find_all(["div", "section", "ul"]):
        count = len(PARCEL_ID.findall(container.get_text(" ", strip=True)))
        if count > best_count:
            best, best_count = container, count
    return ("grid", best) if best_count >= 3 else (None, None)

Step 2 — Normalize headers through a synonym map

Once you have the block, resolve its column labels to canonical names with fuzzy matching so Zoning Code, Zoning District, and Zone all map to one field. This is the same reconciliation logic used across attribute normalization rules, applied to column headers.

from difflib import SequenceMatcher

CANONICAL = {
    "parcel_id": {"parcel", "parcel id", "apn", "parcel number"},
    "district": {"zoning", "zoning code", "zoning district", "zone"},
    "acreage": {"acreage", "acres", "lot size", "area"},
}

def map_header(label: str, threshold: float = 0.82) -> str | None:
    """Map a raw header label to a canonical field via synonym + fuzzy match."""
    norm = label.strip().lower()
    for field, synonyms in CANONICAL.items():
        if norm in synonyms:
            return field
    # Fuzzy fallback for near-misses like "zoning distr." or typos.
    for field, synonyms in CANONICAL.items():
        if any(SequenceMatcher(None, norm, s).ratio() >= threshold
               for s in synonyms):
            return field
    return None

Step 3 — Hash the structure and score confidence

Fingerprint the header set so any rename or reorder changes the hash, and compute a per-row confidence from how many cells resolved cleanly. A changed hash raises drift; a low row score routes that row to review instead of trusting it.

import hashlib

def structure_hash(headers: list[str]) -> str:
    """Order-independent hash of the canonical header set."""
    mapped = sorted(f for h in headers if (f := map_header(h)))
    return hashlib.sha256("|".join(mapped).encode()).hexdigest()[:16]

def row_confidence(record: dict, required=("parcel_id", "district")) -> float:
    """Fraction of required fields present and non-empty, plus format check."""
    present = sum(bool(record.get(k)) for k in required) / len(required)
    valid_id = 1.0 if PARCEL_ID.fullmatch(record.get("parcel_id", "")) else 0.0
    return round(0.6 * present + 0.4 * valid_id, 3)

Step 4 — Parse rows with a pandas fallback and schema assert

For real tables, pandas.read_html is the fastest path; wrap it and fall back to manual cell walking for grid <div>s. Assert the schema after mapping so a silent column swap fails loudly instead of writing acreage into the district column, feeding the same schema validation and data quality checks boundary the pipeline depends on.

import io
import pandas as pd

REQUIRED_FIELDS = {"parcel_id", "district"}

def parse_rows(kind: str, block) -> list[dict]:
    """Extract records from a table or grid block into canonical fields."""
    records: list[dict] = []
    if kind == "table":
        try:
            df = pd.read_html(io.StringIO(str(block)))[0]
        except ValueError:
            return records
        colmap = {c: map_header(str(c)) for c in df.columns}
        if not REQUIRED_FIELDS <= {v for v in colmap.values() if v}:
            raise ValueError(f"Schema drift: mapped {set(colmap.values())}")
        df = df.rename(columns=colmap)
        keep = [c for c in df.columns if c in CANONICAL]
        records = df[keep].to_dict("records")
    else:  # grid: walk repeated row containers
        for row in block.select("[class*=row]"):
            cells = [c.get_text(" ", strip=True)
                     for c in row.select("[class*=cell], span, div")]
            text = " ".join(cells)
            pid = PARCEL_ID.search(text)
            if pid:
                records.append({"parcel_id": pid.group(),
                                "district": cells[-1] if cells else ""})
    return records

A short reference for the drift signals and how each routes:

Signal Detection Route
Structure migration find_data_block returns grid where it was table Parse via grid path; alert
Column rename structure_hash differs from baseline Drift alert; pause job
Column reorder header-map still resolves; positions moved Handled by name mapping
Low field coverage row_confidence below threshold Review queue

Step 5 — Orchestrate with baseline comparison and alerting

Compare the live hash to the stored baseline before parsing, alert on mismatch, and split rows by confidence. When drift fires, the run should page a human and pause rather than write suspect data — the same fail-loud posture formalized under error handling and retry logic.

import logging

logger = logging.getLogger("portal_scraper")

def scrape(soup, baseline_hash: str, min_confidence: float = 0.75):
    kind, block = find_data_block(soup)
    if block is None:
        raise RuntimeError("No data block found; portal layout unrecognized")

    headers = [th.get_text(strip=True) for th in block.select("th")]
    live_hash = structure_hash(headers)
    if headers and live_hash != baseline_hash:
        logger.error(f"DRIFT: hash {live_hash} != baseline {baseline_hash}")
        raise RuntimeError("Structure drift detected; job paused for review")

    good, review = [], []
    for record in parse_rows(kind, block):
        (good if row_confidence(record) >= min_confidence else review).append(record)
    logger.info(f"{len(good)} accepted, {len(review)} to review")
    return good, review

Verification & testing

Confirm resilience against drift, not just a clean happy path.

  • Fixture-replay both layouts. Keep a saved copy of the pre-migration table HTML and a post-migration grid HTML, and assert scrape returns the same records from both. This is the single most valuable regression test for drift.
  • Baseline hash is pinned and reviewed. Store baseline_hash in config, and require a human to update it when a legitimate redesign lands. An auto-updating baseline defeats the entire alert.
  • Row counts reconcile. Compare accepted row count against the portal’s own result count or the previous run’s count; a sudden drop of more than a set threshold is drift even if the hash matched.
  • No positional leakage. Assert that swapping two columns in a test fixture does not change any field value, proving extraction is name-anchored, not position-anchored.
  • Confidence distribution is stable. Track the fraction of rows sent to review per run. A spike means a subtle rename the fuzzy matcher half-caught — inspect before the baseline drifts further.

Failure recovery

When drift fires or a block goes missing mid-run, the job must fail safe and stay reproducible.

  • Pause, never overwrite. On a hash mismatch the job halts before writing, preserving the last known-good target table. Suspect data never reaches the parcel fabric.
  • Snapshot the changed HTML. On drift, persist the raw response to a drift_snapshots/ partition with the timestamp and both hashes so a human can diff old versus new layout and update the mapping.
  • Route low-confidence rows to review. Rows below the confidence floor go to a review queue with their raw cells, not to the target table. A reviewer confirms or corrects them, and the correction feeds back into the synonym map.
  • Alert on the mismatch immediately. Emit a structured log with portal_id, baseline_hash, live_hash, and rows_review, and page on-call when a hash mismatch occurs or the review fraction exceeds 5% of the run.
  • Keep the previous mapping until the new one is verified. When a redesign is confirmed, add the new synonyms without deleting the old ones so a partial rollback on the portal’s side does not re-break you.

Frequently asked questions

Why did my scraper return zero rows without any error?

The portal redesign moved the data out of the element your selector targeted — often from a real <table> into CSS-grid <div>s — so the selector matched nothing while the request still returned 200. Anchor on content signals like header text and a parcel-ID regex, and assert on row count rather than HTTP status, so an empty result raises instead of logging success.

When should I use pandas.read_html versus manual parsing?

Use pandas.read_html when the data is in a genuine <table> element; it handles headers and rowspans well and is far less code. Fall back to manual BeautifulSoup or lxml walking when the portal renders grid <div>s, which read_html ignores entirely. Detect which case you are in first, then dispatch.

How does a structure hash catch a column rename?

The hash is computed over the canonicalized header set, so if Zoning Code becomes Zoning District and the synonym map has not been updated yet, the mapped set changes and the hash diverges from the baseline. That mismatch fires the drift alert before any row is written, giving you a chance to update the mapping deliberately.

Won't fuzzy header matching accept a wrong column by accident?

It can if the threshold is too low, which is why every accepted row also gets a confidence score and low-confidence rows route to review rather than straight to the target table. Keep the fuzzy threshold conservative and lean on the structure hash and review queue as the safety net, not the matcher alone.