Data Lineage & Provenance Tracking

An underwriter approves a mixed-use acquisition on the strength of a single field: the parcel’s zoning code reads C-2, so the pro-forma assumes ground-floor retail is by-right. Nine months later a planning appeal turns on whether that C-2 was ever real, and the question lands on your desk — which municipal publication produced it, on what date, and through which version of your mapping logic? If the answer is a shrug, the dataset is not underwriting-grade no matter how clean the geometry looks. Data lineage and provenance tracking is the discipline that makes that question answerable for every value, every time, by binding each analytical output to the exact bytes it came from and the exact transform that produced it. It is the accountability layer of the Municipal Zoning Data Architecture & Compliance Frameworks framework: the other topics in this area validate, model, and map zoning data, but provenance is what lets you reconstruct any figure back to a defensible origin months after the pipeline ran and the source portal changed.

The failure this topic prevents is subtle because nothing crashes. A pipeline that drops provenance still writes plausible zoning codes; it just cannot prove them. When a regulator, a lender’s counsel, or your own QA team asks “reconstruct this,” a system without lineage has to re-scrape a portal whose contents have since moved, re-run code that has since been refactored, and hope the answer matches. It rarely does. The patterns below make reconstruction deterministic rather than archaeological.

Prerequisites and operational context jump to heading

Provenance tracking is a cross-cutting concern, not a stage you bolt onto the end. It only produces trustworthy evidence if a handful of upstream disciplines are already in place, because a lineage hash over garbage inputs simply proves you faithfully preserved garbage.

• Deterministic source acquisition. The bytes you hash must be reproducible. A raw PDF or a paginated API response has to be captured before any parsing, stored verbatim, and never re-fetched to “refresh” a run. If the acquisition layer normalizes whitespace or re-encodes JSON before you see it, your source hash is already lying about what the municipality published.
• A stable transform identity. Each transformation — a scraper regex, a taxonomy lookup, a geometry repair — needs a durable identifier and version. In practice this is a code version (a git commit or released package version) plus a logical step name, so that “transform [email protected]” means exactly one thing forever.
• A published mapping version. Because local codes are translated through the zoning taxonomy mapping tables, the specific version of those tables is a first-class input. When the mapping changes, the same source bytes can legitimately yield a different standardized code, and lineage must record which mapping was in force.
• Validated schema at the boundary. Provenance records are only as trustworthy as the fields they reference. The schema validation & data quality checks layer must guarantee that parcel_id, zoning_code, and jurisdiction conform before a lineage hash is stamped, or you will content-address inconsistent keys.
• Append-only storage you cannot silently edit. The audit log needs a substrate that resists in-place mutation: a WORM object bucket, a table with revoked UPDATE/DELETE grants, or a periodically anchored hash chain. If an operator can quietly rewrite history, the log proves nothing.

Treat these as gates. If the acquisition layer is non-deterministic or the schema is unvalidated, fix that before investing in lineage, because provenance amplifies the credibility of whatever it wraps — including a credible-looking mistake.

Architecture: content addressing and the chained lineage hash jump to heading

The core idea borrows from version-control systems: identify data by the hash of its content rather than by a mutable path or a database row id. A row id tells you where a value sits today; a content address tells you what the value is, and it never changes as long as the content does not. Provenance tracking layers three ideas on top of that primitive.

Canonicalization comes first. Two byte streams that a human considers “the same JSON” — one pretty-printed, one minified, keys in different order — hash to different values. Before hashing you must reduce inputs to a canonical form: sort object keys, fix the encoding to UTF-8, normalize line endings, and for PDFs hash the raw file bytes rather than extracted text. Without a canonicalization contract, a re-run that merely re-serializes the same data appears to change the source, and every lineage assertion becomes noise.

The lineage hash chains inputs, not just the output. A naive approach hashes the final zoning value and stops. That proves the value’s identity but nothing about its derivation — two different sources could produce the same C-2. Instead, compute the lineage hash by concatenating the canonical source content address, the transform identifier, and the mapping version in a fixed order, then hashing the concatenation. The result is a single stable token that changes if and only if any input to the derivation changed. Chaining also composes: a multi-step pipeline feeds each stage’s lineage hash in as the “source” of the next, so the final token transitively covers the entire path from municipal publication to stored value, the way a Merkle chain covers a history.

The audit record and manifest are the two durable sinks. The append-only audit log is the temporal record — an immutable row per derivation capturing who ran it, when, against which source, and what token resulted. The content-addressed manifest is the spatial record — a store keyed by hash where the canonical source bytes and the transform parameters can be fetched back by their address. Together they answer both “what happened over time” and “give me the exact input again.” The SVG above shows both branching off the stamped record; in production they are frequently the same physical store with two indexes.

This architecture deliberately keeps the value and its proof separable. The zoning code lives in your operational table for queries; the lineage token is a foreign key into an evidence store that a reviewer, not a query planner, consults. That separation is what lets the operational database stay fast while the audit trail stays complete.

Production implementation jump to heading

The module below implements the three artifacts the architecture calls for: a deterministic lineage hash that chains a source content address with a transform identity and mapping version, an append-only audit record modeled with pydantic, and a content-addressed manifest writer. It is written to be dropped into an ingestion worker and called once per derived record. Error handling favours failing closed — a record that cannot be provenanced is quarantined rather than stored without lineage.

import hashlib
import json
import logging
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Optional

from pydantic import BaseModel, Field, ValidationError, field_validator

logger = logging.getLogger("zoning.lineage")

# Version this constant. If the canonicalization or chaining rule ever changes,
# bump it so old and new tokens are never mistaken for one another.
LINEAGE_ALGO_VERSION = "lineage-v1"


class LineageError(Exception):
    """Raised when a record cannot be provenanced and must be quarantined."""


def canonical_bytes(payload: Any) -> bytes:
    """Reduce a value to a stable byte form so equal content hashes equally.

    Dicts/lists are serialized with sorted keys and no incidental whitespace;
    raw bytes (e.g. an original PDF) pass through untouched so we hash exactly
    what the municipality published.
    """
    if isinstance(payload, bytes):
        return payload
    try:
        return json.dumps(
            payload, sort_keys=True, separators=(",", ":"), ensure_ascii=False
        ).encode("utf-8")
    except (TypeError, ValueError) as exc:
        raise LineageError(f"payload is not canonicalizable: {exc}") from exc


def content_address(payload: Any) -> str:
    """Return the sha256 content address of a canonicalized payload."""
    return "sha256:" + hashlib.sha256(canonical_bytes(payload)).hexdigest()


def lineage_hash(
    source_address: str,
    transform_id: str,
    mapping_version: str,
) -> str:
    """Deterministic lineage token chaining source, transform, and mapping.

    The three inputs are joined with a delimiter that cannot appear inside them
    (a NUL byte) so that ("a", "bc") and ("ab", "c") can never collide, then
    hashed together with the algorithm version.
    """
    if not (source_address and transform_id and mapping_version):
        raise LineageError("source, transform_id, and mapping_version are required")
    parts = [LINEAGE_ALGO_VERSION, source_address, transform_id, mapping_version]
    joined = "\x00".join(parts).encode("utf-8")
    return "lin:" + hashlib.sha256(joined).hexdigest()


class LineageRecord(BaseModel):
    """One immutable audit row binding a target value to its exact origin."""

    lineage_id: str
    parcel_id: str
    target_field: str            # e.g. "zoning_code"
    target_value: str            # e.g. "C-2"
    source_address: str          # content address of the canonical source bytes
    transform_id: str            # e.g. "[email protected]"
    mapping_version: str         # e.g. "[email protected]"
    jurisdiction: str
    recorded_by: str             # service account or operator
    recorded_at: datetime = Field(default_factory=lambda: datetime.now(timezone.utc))

    @field_validator("lineage_id")
    @classmethod
    def _prefixed(cls, v: str) -> str:
        if not v.startswith("lin:"):
            raise ValueError("lineage_id must be a lineage token")
        return v

    model_config = {"frozen": True}  # reject in-place mutation after creation


def build_record(
    *,
    parcel_id: str,
    target_field: str,
    target_value: str,
    source_payload: Any,
    transform_id: str,
    mapping_version: str,
    jurisdiction: str,
    recorded_by: str,
) -> LineageRecord:
    """Compute the lineage token and assemble the immutable audit record."""
    src_addr = content_address(source_payload)
    lin = lineage_hash(src_addr, transform_id, mapping_version)
    try:
        return LineageRecord(
            lineage_id=lin,
            parcel_id=parcel_id,
            target_field=target_field,
            target_value=target_value,
            source_address=src_addr,
            transform_id=transform_id,
            mapping_version=mapping_version,
            jurisdiction=jurisdiction,
            recorded_by=recorded_by,
        )
    except ValidationError as exc:
        # Fail closed: never store a target value we cannot fully provenance.
        raise LineageError(f"invalid lineage record for {parcel_id}: {exc}") from exc


class ContentAddressedManifest:
    """Filesystem-backed manifest: source bytes fetchable by their address.

    In production this is an object store (S3/GCS) with object-lock/WORM; the
    local variant here keeps the same content-addressed, write-once contract.
    """

    def __init__(self, root: str) -> None:
        self.root = Path(root)
        self.root.mkdir(parents=True, exist_ok=True)

    def _path(self, address: str) -> Path:
        digest = address.split(":", 1)[-1]
        # Shard by prefix to keep directories small at scale.
        return self.root / digest[:2] / digest

    def put(self, source_payload: Any) -> str:
        """Store canonical bytes under their address; writes are idempotent."""
        address = content_address(source_payload)
        path = self._path(address)
        if path.exists():
            return address  # already stored — content addressing is write-once
        path.parent.mkdir(parents=True, exist_ok=True)
        tmp = path.with_suffix(".tmp")
        try:
            tmp.write_bytes(canonical_bytes(source_payload))
            tmp.replace(path)  # atomic publish; readers never see a partial file
        except OSError as exc:
            tmp.unlink(missing_ok=True)
            raise LineageError(f"manifest write failed for {address}: {exc}") from exc
        return address

    def get(self, address: str) -> bytes:
        """Fetch the exact source bytes behind a lineage record for review."""
        path = self._path(address)
        if not path.exists():
            raise LineageError(f"manifest miss: {address} not found")
        return path.read_bytes()


def provenance_stamp(
    manifest: ContentAddressedManifest,
    audit_sink,  # callable persisting one LineageRecord append-only
    *,
    parcel_id: str,
    target_value: str,
    source_payload: Any,
    transform_id: str,
    mapping_version: str,
    jurisdiction: str,
    recorded_by: str = "ingestion-worker",
    target_field: str = "zoning_code",
) -> str:
    """Stamp one record: store source, build audit row, return lineage token."""
    manifest.put(source_payload)  # ensure the source is fetchable by address
    record = build_record(
        parcel_id=parcel_id,
        target_field=target_field,
        target_value=target_value,
        source_payload=source_payload,
        transform_id=transform_id,
        mapping_version=mapping_version,
        jurisdiction=jurisdiction,
        recorded_by=recorded_by,
    )
    try:
        audit_sink(record)  # append-only; must reject updates/deletes
    except Exception as exc:
        logger.error("audit append failed for %s: %s", parcel_id, exc)
        raise LineageError(f"could not persist audit row for {parcel_id}") from exc
    logger.info("provenanced %s %s=%s -> %s",
                parcel_id, target_field, target_value, record.lineage_id)
    return record.lineage_id

Several choices matter for defensibility. Joining the chained inputs with a NUL delimiter closes the ambiguity where different splits of the same characters could otherwise collide. The LINEAGE_ALGO_VERSION prefix means that if you ever revise canonicalization, old tokens remain valid and are never confused with new ones. Marking LineageRecord as frozen enforces at the object level the same immutability the storage layer enforces at rest. And provenance_stamp writes the source to the manifest before it appends the audit row, so a crash never leaves an audit record pointing at bytes that were never stored.

Edge cases and gotchas jump to heading

Provenance systems fail quietly, in ways that only surface during the audit you built them for. Guard against these specifically:

• Non-deterministic canonicalization. Floating-point coordinates re-serialized with different precision, or a datetime rendered with a local timezone, will change a source address without any real change in content. Pin numeric formatting and normalize all timestamps to UTC before hashing.
• Hashing extracted text instead of raw bytes. For PDFs, hash the original file, not the string your extractor produced. Extraction is part of the transform, so folding it into the source address makes the proof depend on your parser version and defeats the purpose of separating source from transform.
• Silent mapping-version drift. If the taxonomy tables are loaded from a mutable file with no version stamp, two runs a week apart record the same mapping_version string while using different tables. Content-address the mapping tables themselves and derive the version from that hash.
• Truncated or “short” hashes. Storing the first 8 hex characters to save space invites collisions at scale and makes reconstruction ambiguous. Keep the full digest; storage of a 64-character string is not your bottleneck.
• Audit rows that reference deleted sources. A retention policy that expires manifest objects after 90 days will orphan every audit row older than that. Provenance retention must be the longest retention in the system, tied to the underwriting or regulatory window, not the operational one.
• Backfills that overwrite lineage. When you reprocess history with an improved transform, do not mutate existing lineage records. Append new ones with the new transform_id; the old tokens remain the truthful record of what the earlier pipeline actually produced.

Reference: lineage artifacts and what each one proves jump to heading

Each artifact answers a distinct audit question. Confusing them — for instance, treating a content address as proof of derivation rather than content — is where provenance arguments fall apart under scrutiny.

Integration points jump to heading

Provenance is defined by what it wraps. Its inputs are the standardized codes emitted by zoning taxonomy mapping, whose table version becomes a first-class lineage input — change the mapping and the lineage hash legitimately changes, which is the behaviour you want. Its trust depends on schema validation & data quality checks having certified the fields before they are stamped, so that the audit row references keys that actually mean what they claim. And its downstream consumer is change analysis: the temporal versioning & snapshots layer records how a parcel’s zoning evolved over time, and each snapshot it stores carries the lineage token of the value that produced it, so a detected change can always be traced to the two source publications that differed. The contract in every direction is the lineage token itself: an opaque, stable string that any other topic can carry as a foreign key without understanding how it was computed.

Compliance and audit artifacts jump to heading

For PropTech underwriting and regulatory review, provenance is the difference between a dataset and admissible evidence. A defensible run emits and retains:

• A per-value lineage token stored alongside every zoning code in the operational table, so no field exists without a pointer to its origin.
• An append-only audit log with one immutable row per derivation, carrying the actor, timestamp, jurisdiction, transform version, and mapping version — the record that survives even after the operational value is later corrected.
• A content-addressed manifest holding the canonical source bytes, retained for the full regulatory window, so a reviewer can fetch the exact PDF or API response the value came from.
• A reconstruction procedure that, given any lineage token, re-fetches the source, re-runs the pinned transform and mapping, and asserts the recomputed token matches — the test that turns “we think this is right” into “this reproduces bit-for-bit.”

These artifacts slot directly into the broader compliance framework integration layer, which maps them onto whatever attestation a given lender, jurisdiction, or auditor requires. Provenance produces the evidence; compliance integration presents it in the shape the reviewer expects.

FAQ jump to heading

Related jump to heading

• Stamping source-to-target lineage hashes on every record — the step-by-step build for retrofitting provenance onto un-stamped records
• Schema Validation & Data Quality Checks — certifies the fields before they are provenanced
• Zoning Taxonomy Mapping — supplies the mapping version that lineage records as a first-class input
• Compliance Framework Integration — presents lineage evidence in the shape each auditor requires
• Temporal Versioning & Snapshots — the downstream consumer that carries lineage tokens through change history

Up: Municipal Zoning Data Architecture & Compliance Frameworks