Geospatial Format Conversion

Municipal spatial data arrives in whatever format a county’s GIS shop settled on a decade ago, and it has to leave in whatever format your analytics stack, tiling server, or columnar warehouse expects today. In between sits a conversion step that looks trivial and quietly destroys data. A parcel layer downloaded as an ESRI Shapefile carries a coordinate reference system in a sidecar .prj, attribute types in a .dbf, and encoding assumptions nobody wrote down; push it through a careless to_file() call and the field named EFFECTIVE_ZONING_DISTRICT becomes EFFECTIVE_Z, the projection collapses to an unlabeled blob, and a float acreage column silently rounds to an integer. This topic is part of the Automated Feed Ingestion & GIS Data Parsing framework, and it governs the seam where ingested, validated features are re-serialized for downstream consumers — the point at which lossy conversion introduces errors that no amount of later validation can recover, because the original information is already gone.

The goal is not merely to change formats but to change them losslessly: every coordinate reference system stays authoritative and machine-readable, every field keeps its name and declared type, and every geometry survives the round trip topologically valid. That discipline is what separates a conversion utility from a data-corruption engine.

Prerequisites and operational context jump to heading

Format conversion is a boundary operation, and boundaries are where assumptions leak. Before the patterns below hold up in production, several things must already be settled upstream:

• A resolved, authoritative CRS on every input. Conversion cannot invent a projection that the source never declared. Inputs that arrive with a missing or ambiguous .prj must first pass through the CRS alignment strategies you standardise on, so that by the time a layer reaches the converter its EPSG code is known and trusted rather than guessed at write time.
• Normalised attribute names and types. The converter should not be the place where a county’s ZONE_CD and another’s zoning_code get reconciled. That belongs to your attribute normalization rules; the converter’s job is to preserve whatever schema it is handed, not to redesign it mid-flight.
• Validated geometry. GEOS-invalid polygons — self-intersections, unclosed rings, zero-area slivers — behave differently across drivers. GeoPackage tolerates some invalidity that GeoParquet’s stricter geometry metadata will reject. Repair or quarantine before conversion, not after.
• A declared target contract. Know why you are converting. A tiling server wants GeoJSON in EPSG:4326; a columnar analytics job wants GeoParquet partitioned by county; a desktop QGIS handoff wants a single multi-layer GeoPackage. The target format is a downstream requirement, not a preference.
• GDAL parity across environments. pyogrio, fiona, and geopandas all bind to a system GDAL/OGR build. A converter that works on your laptop and fails in CI is almost always a GDAL version skew — GeoParquet and FlatGeobuf drivers only exist in GDAL 3.5+ and later. Pin the library stack.

With those in place, conversion becomes a deterministic transform. Without them, it becomes the stage that launders bad data into an authoritative-looking output.

Architecture: convert through a canonical representation, never format-to-format jump to heading

The single most damaging conversion anti-pattern is chaining formats directly — Shapefile to GeoJSON to GeoParquet — because each hop applies the most restrictive rules of the intermediate format to everything that passes through it. Round-trip a layer through GeoJSON and every geometry is now WGS84 with truncated coordinate precision and every field type has collapsed to string or number, regardless of what the eventual GeoParquet target could have preserved. The intermediate format’s limitations become permanent.

The correct architecture reads each source exactly once into a single canonical in-memory representation — in the Python ecosystem, a geopandas.GeoDataFrame backed by pyogrio — that holds three things the naive path throws away:

• The CRS as a pyproj.CRS object, not a filename or a loose WKT string. This is an authoritative, EPSG-resolvable identity that every writer can embed in its own native metadata slot.
• Per-column dtypes, so an integer parcel count stays integer, a nullable acreage stays nullable float, and a date field is not silently stringified.
• Geometry objects with validated topology, so the writer serialises known-good rings rather than re-parsing text.

From that canonical table, you write to any target independently. GeoParquet writes go through Arrow with the CRS embedded in the geo column metadata per the OGC GeoParquet spec; GeoPackage writes go through OGR into a SQLite container that can hold many layers and a full spatial reference table; FlatGeobuf writes a single streamable, spatially-indexed binary; GeoJSON writes reproject to WGS84 last, as a deliberate lossy export for the web, never as an intermediary.

The field-name problem deserves its own mechanism. The DBF component of a Shapefile hard-limits field names to ten characters, so any layer that originated as a Shapefile may already carry truncated, collision-prone names (ZONING_DIS, ZONING_DI2). When you convert away from Shapefile into a format with no such limit, you want the full, meaningful names back; when you convert into Shapefile, you must control the truncation deterministically rather than letting OGR mangle it. Both directions require an explicit, persisted field-name map that lives alongside the data — a small dictionary translating storage-safe names to canonical names — so the round trip is reversible and auditable.

Production implementation jump to heading

The converter below reads any OGR-supported source into a canonical GeoDataFrame, preserves the CRS and schema, applies a bidirectional field-name map to survive DBF truncation, validates geometry, and writes to the requested target with the CRS embedded natively. It is built on pyogrio for speed (it reads and writes through GDAL’s Arrow interface) and falls back cleanly when a driver or dependency is unavailable.

import json
import logging
from pathlib import Path
from typing import Dict, Optional

import geopandas as gpd
import pyogrio
from pyproj import CRS
from shapely.validation import make_valid

logger = logging.getLogger("format_conversion")

# Drivers keyed by canonical target name -> (OGR/pyogrio driver, file suffix).
TARGET_DRIVERS = {
    "geoparquet": ("Parquet", ".parquet"),
    "flatgeobuf": ("FlatGeobuf", ".fgb"),
    "geopackage": ("GPKG", ".gpkg"),
    "geojson": ("GeoJSON", ".geojson"),
    "shapefile": ("ESRI Shapefile", ".shp"),
}

# GeoJSON is WGS84-only by spec (RFC 7946); everything else keeps source CRS.
WGS84_ONLY = {"geojson"}


def _resolve_source_crs(gdf: gpd.GeoDataFrame, fallback_epsg: Optional[int]) -> CRS:
    """Return an authoritative pyproj.CRS or raise; never write an unlabeled CRS."""
    if gdf.crs is not None:
        return CRS.from_user_input(gdf.crs)
    if fallback_epsg is not None:
        logger.warning("Source CRS missing; applying declared fallback EPSG:%s", fallback_epsg)
        return CRS.from_epsg(fallback_epsg)
    raise ValueError(
        "Source has no CRS and no fallback_epsg was provided. Refusing to write "
        "a layer with an unlabeled projection — resolve the CRS upstream first."
    )


def _validate_geometry(gdf: gpd.GeoDataFrame) -> gpd.GeoDataFrame:
    """Repair invalid geometry in place; quarantine anything still empty/null."""
    invalid = ~gdf.geometry.is_valid
    if invalid.any():
        logger.info("Repairing %d invalid geometries with make_valid", int(invalid.sum()))
        gdf.loc[invalid, gdf.geometry.name] = gdf.loc[invalid, gdf.geometry.name].apply(make_valid)
    keep = gdf.geometry.notna() & ~gdf.geometry.is_empty
    dropped = int((~keep).sum())
    if dropped:
        logger.warning("Quarantining %d empty/null geometries before write", dropped)
    return gdf.loc[keep].copy()


def _apply_field_map(gdf: gpd.GeoDataFrame, target: str,
                     field_map: Optional[Dict[str, str]]) -> gpd.GeoDataFrame:
    """Rename columns for the target, returning storage-safe names for Shapefile.

    field_map is canonical_name -> short_name. For Shapefile we rename to the
    short (<=10 char) form; for every other target we keep the canonical names.
    """
    if not field_map:
        return gdf
    if target == "shapefile":
        renamed = gdf.rename(columns=field_map)
        clashes = [c for c in renamed.columns if len(c) > 10]
        if clashes:
            raise ValueError(f"field_map left names >10 chars for Shapefile: {clashes}")
        return renamed
    # Converting away from Shapefile: invert the map to restore full names.
    inverse = {short: canon for canon, short in field_map.items()}
    return gdf.rename(columns={c: inverse.get(c, c) for c in gdf.columns})


def convert_layer(
    source_path: str,
    target: str,
    out_path: str,
    layer: Optional[str] = None,
    fallback_epsg: Optional[int] = None,
    field_map: Optional[Dict[str, str]] = None,
) -> Dict[str, object]:
    """Read one layer and write it losslessly to `target`, preserving CRS + schema."""
    target = target.lower()
    if target not in TARGET_DRIVERS:
        raise ValueError(f"Unsupported target '{target}'. Known: {list(TARGET_DRIVERS)}")
    driver, suffix = TARGET_DRIVERS[target]

    # 1. Read once into the canonical table (pyogrio uses GDAL's Arrow path).
    try:
        gdf = gpd.read_file(source_path, layer=layer, engine="pyogrio")
    except Exception as exc:
        logger.error("Failed to read %s: %s", source_path, exc)
        raise

    original_dtypes = {c: str(gdf[c].dtype) for c in gdf.columns if c != gdf.geometry.name}

    # 2. Establish an authoritative CRS and validate geometry.
    source_crs = _resolve_source_crs(gdf, fallback_epsg)
    gdf = gdf.set_crs(source_crs, allow_override=True)
    gdf = _validate_geometry(gdf)

    # 3. GeoJSON is WGS84-only; reproject last, and only for that target.
    if target in WGS84_ONLY and source_crs.to_epsg() != 4326:
        logger.info("Reprojecting to EPSG:4326 for GeoJSON export (lossy, web-facing)")
        gdf = gdf.to_crs(epsg=4326)

    # 4. Apply the field-name map so DBF truncation is deterministic + reversible.
    gdf = _apply_field_map(gdf, target, field_map)

    # 5. Write with the CRS embedded in the target's native metadata slot.
    out = Path(out_path).with_suffix(suffix)
    out.parent.mkdir(parents=True, exist_ok=True)
    try:
        if target == "geoparquet":
            # geopandas embeds CRS + geometry types in the GeoParquet 'geo' metadata.
            gdf.to_parquet(out, index=False)
        else:
            pyogrio.write_dataframe(gdf, out, driver=driver)
    except Exception as exc:
        logger.error("Write to %s (%s) failed: %s", out, driver, exc)
        raise

    # 6. Emit a sidecar manifest so the conversion is auditable + reversible.
    manifest = {
        "source": str(source_path),
        "target_format": target,
        "output": str(out),
        "crs_epsg": (gdf.crs.to_epsg() if gdf.crs else None),
        "crs_wkt": (gdf.crs.to_wkt() if gdf.crs else None),
        "feature_count": int(len(gdf)),
        "source_dtypes": original_dtypes,
        "field_map": field_map or {},
    }
    out.with_suffix(suffix + ".manifest.json").write_text(json.dumps(manifest, indent=2))
    logger.info("Converted %s -> %s (%d features, EPSG:%s)",
                source_path, out, len(gdf), manifest["crs_epsg"])
    return manifest

A few decisions in this converter are deliberate. _resolve_source_crs refuses to write an unlabeled projection rather than defaulting to WGS84 — silently assuming EPSG:4326 is how parcels end up in the Gulf of Mexico. Reprojection happens only for the GeoJSON target and only as the final step, so the canonical table keeps the source CRS for every other output. The field-name map is bidirectional: it truncates deterministically when writing Shapefile and restores full names when reading one back out. And every conversion drops a sidecar manifest recording the source, the resolved EPSG, per-column dtypes, and the field map, so the operation is both auditable and reversible.

Edge cases and gotchas jump to heading

Format conversion fails in format-specific ways. These recur across municipal datasets:

• DBF character encoding. Shapefiles predate universal UTF-8, and a .dbf written in Windows-1252 or an East-Asian code page will mojibake a street name like Peña Blvd unless you read it with the matching encoding (a .cpg sidecar declares it when present). Force the encoding explicitly; do not trust the default.
• The 2 GB Shapefile ceiling. Both the .shp and .dbf components are capped at roughly 2 GB by their 32-bit offset headers. A large county parcel fabric will silently truncate at the boundary — features past the limit simply vanish with no error. This alone is a reason to convert out of Shapefile into GeoParquet or GeoPackage for anything large.
• GeoJSON coordinate precision. RFC 7946 mandates WGS84 and, in practice, tools emit full double-precision coordinates that bloat file size. Rounding to six decimal places (~11 cm) is usually safe for parcels, but rounding is lossy and must be a conscious choice, not a driver default you discover later.
• Mixed geometry types. A GeoPackage layer happily mixes Polygon and MultiPolygon; a Shapefile forces one geometry type per file and will either promote everything to multi or split the layer. Normalise to MultiPolygon before a Shapefile write to avoid a surprise.
• Null-vs-empty geometry. Some drivers write a null geometry, others an empty one, and they are not interchangeable downstream — a spatial index treats them differently. Decide on one convention and enforce it in the validation step, as the converter above does.
• Three-dimensional coordinates. A source with Z values (elevation, or a stray 0.0 third ordinate) converts fine to GeoPackage but may confuse a 2D-only consumer. Flatten to 2D explicitly if the downstream contract is planar.
• CRS axis order. EPSG defines some CRSs as latitude-first. A converter that ignores axis order can swap X and Y on reprojection; always drive pyproj transforms with always_xy=True so easting/northing order is unambiguous.

Format reference jump to heading

Choose the target by what the downstream consumer needs, then convert once through the canonical table. The trade-offs:

Integration points jump to heading

Conversion is defined by what precedes and follows it. Its inputs are validated, normalised features; its outputs are the serialised artifacts other systems consume. The most direct downstream consumer is GIS export sync workflows, which schedule and publish these converted files to portals, object storage, and tile servers — the converter produces the bytes, the sync workflow moves and versions them. Upstream, the schema the converter must preserve is set by attribute normalization rules; if two counties disagree on field names, that disagreement is resolved before conversion, and the converter’s field-name map only handles the mechanical DBF-truncation problem, never the semantic reconciliation. The CRS the converter treats as authoritative is whatever CRS alignment strategies resolved during ingestion, which is why the converter refuses to invent one. Cleanly converted, CRS-stable outputs are also what make downstream geometry diffing and change detection trustworthy — a comparison across two GeoParquet snapshots only means something when both were written from the same authoritative projection.

Compliance and audit artifacts jump to heading

For PropTech underwriting and municipal record-keeping, a converted file is only defensible if the conversion is provable. Every conversion run should emit:

• A conversion manifest — the source path and hash, target format and driver version, the resolved EPSG and full CRS WKT, feature count in and out, and the field-name map, exactly as the implementation above writes. This is the lineage record linking an output artifact back to the exact input and code.
• The preserved schema — the original per-column dtypes captured before write, so a reviewer can confirm no type collapsed silently during conversion.
• A geometry-integrity summary — counts of geometries repaired and quarantined, so topological changes are recorded rather than absorbed.
• A reversibility guarantee — because the field-name map is persisted, a Shapefile round trip is provably reversible; the manifest is the proof.

These artifacts pair with downstream schema validation & data quality checks to keep the chain from municipal publication to warehouse-grade dataset continuous and defensible, with no lossy hop hiding inside the format step.

FAQ jump to heading

Related jump to heading

• Converting shapefiles to GeoParquet without losing CRS metadata — the null-CRS failure mode diagnosed and repaired step by step
• GIS export sync workflows — the downstream consumer that publishes converted artifacts
• Attribute normalization rules — resolves the field semantics the converter preserves
• CRS alignment strategies — establishes the authoritative CRS the converter treats as truth
• Schema validation & data quality checks — verifies the converted schema downstream

Up: Automated Feed Ingestion & GIS Data Parsing