Scraping zoning PDFs with Python and PyPDF2

You run a county’s latest batch of zoning amendments through PdfReader(path).pages[0].extract_text() and get back (cid:12)(cid:9)(cid:45) instead of a district code, or a setback bearing that reads 00.052 .N E with the digits transposed. The PDFs open fine in a viewer, so the data is there — but PyPDF2 returns garbled bytes, empty strings, or coordinates in the wrong order. This page answers one narrow question: how do you reliably pull zoning district codes and parcel coordinate references out of non-standardized municipal PDFs when the character mappings, text rotation, and font encodings are broken? It is a focused, runnable companion to the broader PDF & HTML scraping pipelines that feed the wider automated zoning change and municipal GIS tracking workflow. The fix is a fallback extraction chain, deterministic coordinate parsing with explicit projection, and a per-document audit artifact you can roll back to.

Diagnosis: identifying why extraction returns garbage jump to heading

PyPDF2 — now merged into the import-compatible successor pypdf (from pypdf import PdfReader) — translates character codes through the PDF’s internal /ToUnicode CMap. Municipal GIS departments routinely export zoning documents from legacy CAD-to-PDF converters that strip or corrupt that mapping, so the same call that works on a born-digital ordinance fails silently on a plat map. The corruption shows up as three distinct, recognizable symptoms.

1. CMap / encoding drift. The page uses Identity-H or a custom WinAnsi encoding with no Unicode table, so extract_text() returns (cid:NN) tokens or mojibake. PyPDF2 may emit a PdfReadWarning; more often it returns a string that is structurally fine but semantically meaningless, which is far more dangerous because it passes a naive if text: check and corrupts downstream.
2. Rotated and reordered text streams. Title blocks, north arrows, and margin metadata are drawn with a rotation matrix. PyPDF2 reads glyphs in stream order, not visual order, so a 90°-rotated bearing label arrives with its characters interleaved or reversed — N 45° 30' E becomes E '03 °54 N.
3. Vector-only geometry. Parcel boundaries, easement polygons, and setback lines are frequently embedded as vector paths, not selectable text. extract_text() ignores them entirely, returning a page that looks empty even though it is dense with spatial primitives.

Reproduce the failure deterministically before changing anything, and record which symptom you have so the fallback can be measured:

from pypdf import PdfReader

reader = PdfReader("zoning_amendment.pdf")
page = reader.pages[0]
text = page.extract_text() or ""

print(f"chars={len(text)}  rotation={page.get('/Rotate', 0)}")
print(f"cid_tokens={text.count('(cid:')}")
print(repr(text[:200]))

A non-zero cid_tokens count or an empty text on a visibly populated page confirms an encoding or vector-only problem that no PyPDF2 parameter can fix — extract_text() accepts no encoding or error-handling arguments and reads the embedded mappings as-is. The only reliable path is a fallback chain.

Step-by-step implementation jump to heading

Each step isolates one concern. Compose them into the orchestrator in the final step.

Step 1 — Build a fallback extraction chain jump to heading

pdfplumber runs on pdfminer.six and handles a wider range of encoding edge cases than PyPDF2; pymupdf (fitz) gives the most robust coverage for CAD-derived PDFs and also recovers rotated text in visual order. Try the fastest tool first, fall back on empty or cid-laden output, and log which tool won so you can track parsing drift per source.

import re
import logging
from pathlib import Path
from typing import List

logger = logging.getLogger("zoning_ingestion")

ZERO_WIDTH = re.compile(r"[​-‍]")
CID_TOKEN = re.compile(r"\(cid:\d+\)")

def _is_usable(text: str) -> bool:
    """Reject empty or cid-token-laden output that survived a naive truthiness check."""
    if not text or not text.strip():
        return False
    # More than ~5% cid tokens means the CMap is corrupt; fall through.
    return len(CID_TOKEN.findall(text)) / max(len(text.split()), 1) < 0.05

def safe_extract_zoning_text(pdf_path: str) -> List[str]:
    """Extract per-page text, falling back pypdf -> pdfplumber -> pymupdf."""
    from pypdf import PdfReader
    import pdfplumber
    import fitz  # pymupdf

    path = Path(pdf_path)
    cleaned: List[str] = []

    try:
        page_count = len(PdfReader(str(path)).pages)
    except Exception as exc:
        logger.error(f"pypdf could not open {path.name}: {exc}")
        return cleaned

    for page_num in range(page_count):
        raw, tool = "", None

        try:
            raw = PdfReader(str(path)).pages[page_num].extract_text() or ""
            tool = "pypdf" if _is_usable(raw) else None
        except Exception as exc:
            logger.warning(f"pypdf failed p{page_num}: {exc}")

        if not tool:
            try:
                with pdfplumber.open(str(path)) as plumb:
                    raw = plumb.pages[page_num].extract_text() or ""
                tool = "pdfplumber" if _is_usable(raw) else None
            except Exception as exc:
                logger.warning(f"pdfplumber failed p{page_num}: {exc}")

        if not tool:
            try:
                # pymupdf "text" sort reorders rotated streams into visual order
                raw = fitz.open(str(path))[page_num].get_text("text", sort=True) or ""
                tool = "pymupdf" if _is_usable(raw) else None
            except Exception as exc:
                logger.error(f"pymupdf failed p{page_num}: {exc}. Skipping page.")

        if not tool:
            logger.error(f"All extractors failed p{page_num} of {path.name}. Quarantining.")
            continue

        normalized = ZERO_WIDTH.sub("", raw)
        normalized = re.sub(r"\s+", " ", normalized).strip()
        if normalized:
            logger.info(f"p{page_num} extracted via {tool}")
            cleaned.append(normalized)

    return cleaned

Step 2 — Recover rotated text in visual order jump to heading

When the diagnostic showed /Rotate is non-zero or bearings arrive scrambled, the pymupdf branch above already helps because get_text("text", sort=True) reconstructs reading order from glyph coordinates. For pages where only a rotated title block matters, target it directly rather than the whole page so unrelated body text is not reordered:

import fitz

def extract_rotated_block(pdf_path: str, page_num: int) -> str:
    """Pull text in geometric reading order, undoing stream-order scrambling."""
    doc = fitz.open(pdf_path)
    page = doc[page_num]
    # words: (x0, y0, x1, y1, "word", block, line, word_no)
    words = page.get_text("words")
    # Sort top-to-bottom, then left-to-right in visual space.
    words.sort(key=lambda w: (round(w[1] / 5), w[0]))
    return " ".join(w[4] for w in words)

Step 3 — Extract coordinate references with a precise regex jump to heading

Zoning legal descriptions carry degree-minute-second (DMS) bearings and decimal-degree pairs. Match them explicitly so a transposed or whitespace-injected string fails loudly instead of parsing into a plausible-but-wrong number.

import re

COORD_PATTERN = re.compile(
    r"(?P<lat>[-+]?\d{1,3}[°\s]\d{1,2}[′']\d{1,2}[″\"]\s*[NS]?)\s*"
    r"(?P<lon>[-+]?\d{1,3}[°\s]\d{1,2}[′']\d{1,2}[″\"]\s*[EW]?)",
    re.IGNORECASE,
)

Step 4 — Enforce explicit CRS transformation jump to heading

Municipal documents reference local State Plane zones in feet while downstream GIS layers expect WGS84, and an implicit datum assumption is the single most common cause of parcels landing in the wrong county. Pin the source EPSG per jurisdiction and transform deterministically — the same discipline covered in depth under CRS alignment strategies, applied here at parse time.

from pyproj import Transformer

def parse_and_transform_coordinates(
    text_block: str,
    source_epsg: int = 2263,   # NY State Plane Long Island (feet)
    target_epsg: int = 4326,
) -> list:
    """Extract DMS pairs and project to the target CRS."""
    transformer = Transformer.from_crs(
        f"EPSG:{source_epsg}", f"EPSG:{target_epsg}", always_xy=True
    )
    parsed = []
    for match in COORD_PATTERN.finditer(text_block):
        try:
            lat_str = re.sub(r"[°′″]", " ", match.group("lat"))
            lon_str = re.sub(r"[°′″]", " ", match.group("lon"))
            lat_deg = float(lat_str.split()[0])
            lon_deg = float(lon_str.split()[0])
            # always_xy=True prevents axis-order inversion (lon, lat) -> (x, y)
            x, y = transformer.transform(lon_deg, lat_deg)
            parsed.append({
                "lon": x, "lat": y,
                "source_epsg": source_epsg, "target_epsg": target_epsg,
            })
        except (ValueError, IndexError):
            continue
    return parsed

A short reference for the State Plane zones you will hit most often when pinning source_epsg:

Step 5 — Emit a deterministic compliance artifact jump to heading

Every parsed document must produce an immutable, schema-validated record that ties the extracted district and coordinates back to the source hash. Enforce it with pydantic, the same boundary that downstream schema validation and data quality checks rely on, and normalize district codes against the zoning taxonomy mapping before serialization.

from datetime import datetime, timezone
from pydantic import BaseModel, Field, ValidationError
import hashlib
import json
import logging

logger = logging.getLogger("zoning_ingestion")

class ZoningComplianceArtifact(BaseModel):
    document_hash: str
    municipality: str
    zoning_district: str
    effective_date: datetime
    coordinates: list
    parsing_metadata: dict = Field(default_factory=dict)

def generate_compliance_artifact(
    cleaned_text: list, coords: list, pdf_path: str,
    municipality: str, zoning_district: str, extractor: str,
) -> str:
    with open(pdf_path, "rb") as f:
        doc_hash = hashlib.sha256(f.read()).hexdigest()
    try:
        artifact = ZoningComplianceArtifact(
            document_hash=doc_hash,
            municipality=municipality,
            zoning_district=zoning_district,
            effective_date=datetime.now(timezone.utc),
            coordinates=coords,
            parsing_metadata={
                "pages_processed": len(cleaned_text),
                "extractor": extractor,
                "coord_count": len(coords),
            },
        )
        return json.dumps(artifact.model_dump(mode="json"), indent=2)
    except ValidationError as exc:
        logger.error(f"Compliance artifact validation failed: {exc}")
        raise

Verification & testing jump to heading

Confirm the fallback chain fixed the failure rather than masking it.

• No surviving cid tokens. Assert "(cid:" not in "".join(cleaned_pages). A leak means _is_usable let corrupt output through — tighten the threshold or force the pymupdf branch for that source.
• Coordinates land in the right county. Cross-reference each transformed point’s bounding box against the municipality’s parcel shapefile. A point that falls in the ocean or an adjacent state is almost always an axis-order or source_epsg mistake, not a regex miss.
• Round-trip the projection. Transform WGS84 back to the source EPSG and assert the result matches the original within tolerance; drift beyond a few feet signals a wrong source zone.
• Deterministic hash. Re-running the same PDF must reproduce an identical document_hash and coord_count. Any change means non-deterministic extraction ordering — pin the extractor per page in the manifest.
• Log signature. A healthy run logs one extracted via <tool> line per page and zero Quarantining lines. Track the ratio of non-pypdf extractions as your parsing-drift metric.

Failure recovery jump to heading

When a page defeats the whole chain mid-batch, the run must continue and stay reproducible.

• Quarantine, never drop. A page that fails all three extractors is written to a failed_pages/ partition with its source hash, page index, /Rotate value, and the last extractor’s exception, then the loop continues. This is the same dead-letter discipline used across error handling and retry logic in the ingestion layer.
• Route scanned PDFs to OCR. If cid_tokens is zero but chars is also zero across every page, the document is a raster scan with no text layer. Flag it for an OCR pass (Tesseract via pdf2image) rather than retrying text extractors that will always return empty.
• Resume from the last page index. Persist the last successfully processed page index per document. On restart, skip pages already present in the manifest so reruns are idempotent and never double-write the target table.
• Backoff on the fetch, not the parse. When PDFs are pulled from a throttled municipal portal, apply exponential backoff with jitter on the download and cap retries before the dead-letter queue — concurrency for large county batches belongs to async batch processing and the portal’s rate limit management layer, not the extractor.
• Alert on drift. Emit structured logs with pdf_hash, page_index, extractor, and coordinate_validation_status, and trigger an alert when non-pypdf extraction or quarantine exceeds 2% of the daily batch volume.

Frequently asked questions jump to heading

Related jump to heading

• Parent topic: PDF & HTML Scraping Pipelines
• Section overview: Automated Feed Ingestion & GIS Data Parsing
• CRS Alignment Strategies — pinning source EPSG and projecting parsed coordinates
• Schema Validation & Data Quality Checks — validating the compliance artifact downstream
• Error Handling & Retry Logic — dead-letter and checkpoint patterns for failed pages
• Async Batch Processing — concurrency for large county PDF batches

For authoritative CRS definitions and transformation matrices, see the OGC Well-Known Text representation of Coordinate Reference Systems specification.