ContextQMD
Back to Claude Scientific Skills

Vector I/O, GeoParquet, Arrow, and PostGIS

skills/geopandas/references/data-io.md

2.55.011.9 KB
Original Source

Vector I/O, GeoParquet, Arrow, and PostGIS

GeoPandas 1.x defaults to pyogrio for vector-file I/O. pyogrio and Fiona are bindings to GDAL/OGR; actual formats, options, and behavior depend on the installed native GDAL and drivers.

Local-only intake policy

Do not automatically pass any of these to GeoPandas/GDAL:

  • http://, https://, s3://, gs://, Azure, or another remote URI;
  • GDAL /vsicurl/, /vsis3/, /vsizip/, chained virtual filesystems, or pyogrio zip+... paths;
  • ZIP/KMZ/TAR/GZ/7z/RAR or nested archives;
  • macro-enabled office files or an untrusted permissive driver;
  • a user-supplied PostGIS connection string.

GeoPandas officially supports URLs and GDAL supports network/archive virtual filesystems, but that capability crosses network, decompression, parser, and credential trust boundaries. Obtain explicit approval, verify source/hash and size out of band, unpack in a sandbox with resource limits, then process an allowlisted local regular file.

The bundled CLIs reject URL/VSI/archive syntax, symlinks, path traversal, and non-allowlisted suffixes.

Inspect before reading features

For GDAL-backed formats:

python
from pathlib import Path
import pyogrio

path = Path("approved/input.gpkg")
if not path.is_file() or path.is_symlink():
    raise ValueError("Expected a vetted local regular file")

layers = pyogrio.list_layers(path)
info = pyogrio.read_info(path, layer="approved_layer", force_feature_count=False)
drivers = pyogrio.list_drivers()

Record:

  • primary-file hash and byte size (a Shapefile hash does not cover sidecars);
  • driver, layer, declared feature count, fields/dtypes, encoding, geometry type;
  • CRS and whether bounds were present, but redact precise bounds by default;
  • pyogrio.__gdal_version__, pyogrio.__gdal_geos_version__, Shapely GEOS, pyproj PROJ, package versions, and enabled driver capabilities.

list_drivers() returns capabilities containing r, w, and/or a, but a listed driver is not proof every field/geometry is supported. Treat drivers as an allowlist, not merely a discovered list.

read_file

Stable signature:

python
gdf = geopandas.read_file(
    filename,
    bbox=None,
    mask=None,
    columns=None,
    rows=None,
    engine="pyogrio",
    use_arrow=True,
)

Rules:

  • bbox and mask are mutually exclusive.
  • With pyogrio, a bbox tuple must already be in the dataset CRS. Fiona can reproject a GeoSeries/GeoDataFrame bbox; do not depend on engine-specific implicit behavior.
  • A mask must have an explicit CRS compatible with the source.
  • columns=[] reads geometry without attributes; ignore_geometry=True returns a pandas DataFrame.
  • rows=n reads the first n rows; rows=slice(a, b) reads a slice.
  • where= is evaluated by a driver SQL dialect. Do not concatenate untrusted expressions.
  • Encoding auto-detection can fail; set a verified encoding explicitly.
  • use_arrow=True requires PyArrow and speeds pyogrio bulk transfer, but does not change parser trust or correctness requirements.
  • Bound both file bytes and features. Some drivers cannot cheaply report a count; read at most limit + 1 and fail closed when the limit is exceeded.

GeoPandas may use HTTP range requests or download an entire URL in memory. This skill therefore does not include remote-read examples.

Filters are not always exact

pyogrio documents:

  • bbox/mask coordinates must use the dataset CRS;
  • when GDAL is built with GEOS, geometry intersection filtering is exact;
  • without GEOS, filters can return features whose bounding boxes intersect, requiring a second exact predicate check;
  • Arrow reads involving skip/max may read batches beyond the requested slice before slicing;
  • feature IDs are driver-specific and may start at 0, 1, or another value.

Do not treat driver FID as a portable stable feature ID.

Writing traditional vector formats

Write a new path and reopen it:

python
output = Path("derived/result.gpkg")
if output.exists() or output.is_symlink():
    raise FileExistsError("Choose a new output path")

gdf.to_file(
    output,
    layer="result",
    driver="GPKG",
    engine="pyogrio",
    index=False,
    use_arrow=True,
)

roundtrip = geopandas.read_file(output, layer="result", engine="pyogrio")

Never use implicit overwrite/append. Driver behavior varies and multi-file formats complicate atomic writes.

Format tradeoffs

  • GeoPackage: good general local interchange; multiple layers, one geometry column per layer, SQL-backed metadata.
  • GeoJSON: broadly interoperable but normally WGS84 longitude/latitude, limited type fidelity, text-heavy, and easy to leak precise coordinates.
  • Shapefile: legacy multi-file format with field-name, type, null, encoding, geometry, and size constraints. Avoid for new work.
  • FlatGeobuf: efficient stream/spatial-index format, but interoperability still depends on driver versions.
  • GeoParquet: efficient columnar storage with multiple geometry columns and explicit geospatial metadata.

Traditional formats often cannot store lists, structs, arbitrary objects, or multiple geometry columns. Plan conversions and reject silent loss.

GeoParquet and Feather

python
gdf.to_parquet(
    "derived/result.parquet",
    index=False,
    compression="snappy",
    geometry_encoding="WKB",
    write_covering_bbox=False,
    schema_version="1.0.0",
)

roundtrip = geopandas.read_parquet(
    "derived/result.parquet",
    columns=["feature_id", "geometry"],
)

GeoPandas 1.1.4 write semantics:

  • all geometry columns are preserved;
  • default geometry_encoding="WKB" maximizes interoperability;
  • default supported stable schema is 1.0.0;
  • geometry_encoding="geoarrow" requires GeoParquet 1.1.0, supports single-geometry native encodings, and is still described as experimental;
  • write_covering_bbox=True adds a per-row bbox column and 1.1 covering metadata. It costs compute and may reveal precise extents;
  • schema_version replaces the removed/deprecated old version= usage;
  • index=None writes non-RangeIndex values as columns and stores RangeIndex as metadata. Use an explicit stable feature-ID column instead.

Read semantics:

  • selecting no geometry columns raises; use pandas.read_parquet for a non-spatial result;
  • if the stored primary geometry is omitted, the first selected geometry becomes active;
  • bbox filtering works only when covering metadata/columns were written;
  • if GeoParquet crs metadata is missing, the specification default is OGC:CRS84;
  • an explicit crs: null means unknown/undefined, which is different;
  • WKB/native coordinates are x/y regardless of authority axis order.

GeoParquet 1.1 metadata requires a geo JSON value, primary_column, and metadata for every geometry column. Geometry columns must be root-level and may be optional; native child coordinates cannot contain nulls. edges defaults to planar. Feature identifiers are outside the core specification, so define and document your own stable-ID metadata/column.

Do not use bbox covering for public sensitive-location data without generalization and approval.

Arrow in memory

GeoPandas 1.0 added to_arrow() and from_arrow() using GeoArrow extension types. These improve interchange but do not make an array self-validating: verify extension metadata, CRS, geometry encoding/type, nulls, and active geometry after round-trip. GeoPandas 1.1 adds to_pandas_kwargs controls for non-geometry Arrow conversion.

PostGIS without credential leakage

Required writing dependencies are SQLAlchemy, GeoAlchemy2, and psycopg/psycopg2. Create connections from named secrets without embedding or logging a connection URL:

python
import os
from sqlalchemy import URL, create_engine

db_url = URL.create(
    "postgresql+psycopg",
    username=os.environ["GEOPANDAS_POSTGIS_USER"],
    password=os.environ["GEOPANDAS_POSTGIS_PASSWORD"],
    host=os.environ["GEOPANDAS_POSTGIS_HOST"],
    port=int(os.environ["GEOPANDAS_POSTGIS_PORT"]),
    database=os.environ["GEOPANDAS_POSTGIS_DATABASE"],
)
engine = create_engine(db_url)

Read only those named variables (or secret-manager equivalents). Never print db_url, engine.url, exception payloads containing it, or the broader environment.

Use parameterized values and trusted SQL identifiers:

python
from sqlalchemy import text

query = text(
    "SELECT feature_id, geom FROM approved_schema.features "
    "WHERE category = :category"
)
gdf = geopandas.read_postgis(
    query,
    con=engine,
    geom_col="geom",
    params={"category": approved_category},
    chunksize=10_000,
)

read_postgis infers one CRS from the SRID of the first geometry and assigns it to all rows unless crs= is supplied. Verify all geometries share the expected SRID. With chunksize, it returns an iterator; validate every chunk and enforce a total-row limit.

For writes:

python
with engine.begin() as connection:
    gdf.to_postgis(
        "derived_features",
        con=connection,
        schema="approved_schema",
        if_exists="fail",
        index=False,
        chunksize=10_000,
    )
  • Default to if_exists="fail".
  • replace drops an existing table and is destructive.
  • Validate schema/table/geometry column names against an allowlist; do not interpolate user input.
  • GeoPandas 1.1.2 fixed SQL injection through a geometry-column name; remain on a patched version and still validate identifiers.
  • Use least-privilege database roles and a transaction.

Fiona-to-pyogrio migration

GeoPandas 1.0 changed the default engine from Fiona to pyogrio. Differences include:

  • schema/metadata keywords and driver options;
  • writing attribute-only tables;
  • handling empty geometries and unsupported field types;
  • datetime resolution/timezone behavior;
  • append and encoding behavior;
  • error/warning text and filter behavior.

Set engine= explicitly for reproducibility and test round-trips before migration. Do not assume identical outputs merely because both engines use GDAL.

Export verification and provenance

For every output:

  1. choose a new local path and explicit format/driver/layer;
  2. record source hashes, source layer, stack/native versions, CRS, precision, repair, and transformation choices;
  3. record field names/types/nullability, geometry columns/types, stable ID, index policy, encoding, dimensions, and expected losses;
  4. write, then reopen with an independent code path when feasible;
  5. compare row count, stable-ID set, null/empty/invalid/type counts, CRS, bounds at protected precision, and representative attribute values;
  6. hash the completed artifact and store the audit separately.

Use scripts/vector_inventory.py for redacted intake and scripts/export_plan.py for a non-executing output contract.

Sources (verified 2026-07-23)