DataFrame and Arrow Compatibility

perspective-python accepts a Table constructor argument from any of the common Python columnar data libraries. In all three cases, perspective.table (and Table.update()) consume the input directly — there is no need to serialize to Apache Arrow IPC bytes yourself. However, note is still the most efficient way to bulk load data into Table.

PyArrow

python

import pyarrow as pa
import perspective

arrow_table = pa.table({
    "int": pa.array([1, 2, 3], type=pa.int64()),
    "float": pa.array([1.5, 2.5, 3.5], type=pa.float64()),
    "string": pa.array(["a", "b", "c"], type=pa.string()),
})

table = perspective.table(arrow_table)

The same applies to Table.update():

python

table.update(arrow_table)

If you have Arrow data already in IPC format (e.g. read from disk, received over the wire, or produced by another tool), pass the raw bytes directly — both stream and file formats are auto-detected:

python

with open("data.arrow", "rb") as f:
    table = perspective.table(f.read())

Nested columns

Perspective's data model is flat, so Arrow struct and list columns are normalized on ingest.

A struct column is hoisted into one dotted column per leaf, recursively. A null parent nulls every descendant leaf:

python

arrow_table = pa.table({
    "id": pa.array([1, 2], type=pa.int64()),
    "s": pa.array([{"a": 10}, {"a": 20}], type=pa.struct([("a", pa.int64())])),
})

# Schema is `{"id": "integer", "s.a": "integer"}`
table = perspective.table(arrow_table)

Because the flattened names are ordinary columns, a Table created from an explicit schema accepts nested updates with no further configuration:

python

table = perspective.table({"id": "integer", "s.a": "integer"})
table.update(arrow_table)

A list column is controlled by the list_flatten argument:

"zip" (default) expands a row into one row per list element, repeating its non-list siblings. An empty or null list yields a single row with a null in that column, rather than dropping the row. When a row has more than one list column, their non-empty lengths must match.
"cartesian" expands a row into the product of its list columns' lengths, with an empty or null list counting as a single null element.
"stringify" encodes each list as a JSON array in a single string column, leaving the row count unchanged.

python

arrow_table = pa.table({
    "x": pa.array([1, 2], type=pa.int64()),
    "y": pa.array([[10, 20], [30]], type=pa.list_(pa.int64())),
})

# `{"x": [1, 1, 2], "y": [10, 20, 30]}`
perspective.table(arrow_table)

# `{"x": [1, 2], "y": ["[10,20]", "[30]"]}`
perspective.table(arrow_table, list_flatten="stringify")

Polars

python

import polars as pl
import perspective

df = pl.DataFrame({
    "a": [1, 2, 3, 4, 5],
    "b": ["x", "y", "z", "x", "y"],
})

table = perspective.table(df)

Internally, the DataFrame is converted to a pyarrow.Table before ingestion, so Polars columns inherit the Arrow type mapping above.

See also Perspective Virtual Server support for polars.DataFrame

Pandas

pandas.DataFrame is supported via pyarrow.Table.from_pandas, which dictates behavior including type support — see the pyarrow pandas docs for details on which pandas dtypes round-trip cleanly.

python

from datetime import date, datetime
import numpy as np
import pandas as pd
import perspective

data = pd.DataFrame({
    "int": np.arange(100),
    "float": [i * 1.5 for i in range(100)],
    "bool": [True for i in range(100)],
    "date": [date.today() for i in range(100)],
    "datetime": [datetime.now() for i in range(100)],
    "string": [str(i) for i in range(100)],
})

table = perspective.table(data, index="float")