Data Types

Vaex is a hybrid DataFrame - it supports both numpy and arrow data types. This page outlines exactly which data types are supported in Vaex, and which we hope to support in the future. We also provide some tips on how to approach common data structures.

For some additional insight, you are welcome to look at this post as well.

Supported Data Types in Vaex

In the table below we define:

Supported: a column or expression of that type can exist and can be stored in at least one file format;
Unsupported: a column or expression of that type can currently not live within a Vaex DataFrame, but can supported be added in the future;
Will not support: This datatype will not be supported in Vaex going forward.

Framework	Dtype	Supported	Remarks
Python	`int`	yes	Will be converted to a `numpy` array
Python	`float`	yes	Will be converted to a `numpy` array
Python	`datetime`	not yet
Python	`timedelta`	not yet
Python	`str`	yes	Will be converted to Arrow array
`numpy`	`int8`	yes
`numpy`	`int16`	yes
`numpy`	`int32`	yes
`numpy`	`int64`	yes
`numpy`	`float16`	yes	Operations should be upcast to `float64`
`numpy`	`float32`	yes
`numpy`	`float64`	yes
`numpy`	`datetime64`	yes
`numpy`	`timedelta64`	yes
`numpy`	`object ('O')`	no
`arrow`	`int8`	yes
`arrow`	`int16`	yes
`arrow`	`int32`	yes
`arrow`	`int64`	yes
`arrow`	`float16`	yes	Operations should be upcast to `float64`
`arrow`	`float32`	yes
`arrow`	`float64`	yes
`arrow`	`date32`	yes
`arrow`	`time64`	yes
`arrow`	`time32`	yes
`arrow`	`duration`	yes
`arrow`	`struct`	yes	Can't be exported to HDF5 yet, but possible
`arrow`	`dictionary`	yes
`arrow`	`union`	not yet

General advice on data types in Vaex

Vaex requires that each column or expression be of a single data type, as in the case of databases. Having a column of different data type can result in a data type object, which is not supported, and can also give raise to various problems.

The general advice is to prepare your data to have a uniform data type per column prior to using Vaex with it.

python

import vaex
import numpy as np
import pyarrow as pa

Higher dimensional arrays

Vaex support high dimensional numpy arrays. The one requirement the arrays must have the same shape. Currently DataFrames that contain higher dimensional numpy arrays can only be exported to HDF5. We hope that arrow will add support for this soon, so we can export such columns to the arrow and parquet formats also.

For example:

python

x = np.random.randn(100, 10, 10)
df = vaex.from_arrays(x=x)
df

We can also pass a nested list of lists structure to Vaex. This will be converted on the fly to a numpy ndarray. As before, the condition is that the resulting ndarray must be regular.

For example:

python

x = [[1, 2], [3, 4]]
df = vaex.from_arrays(x=x)
df

If we happen to have a non-regular list of lists, i.e. a list of lists where the inner lists are of different lengths, we first need to convert it to an arrow array before passing it to vaex:

python

x = [[1, 2, 3, 4, 5], [6, 7], [8, 9, 10]]
x = pa.array(x)
df = vaex.from_arrays(x=x)
df

Note the arrow structs and lists can not be exported to HDF5 for the time being.

String support in Vaex

Vaex uses arrow under the hood to work with strings. Any strings passed to a Vaex DataFrame will be converted to an arrow type.

For example:

python

x = ['This', 'is', 'a', 'string', 'column']
y = np.array(['This', 'is', 'one', 'also', None])

df = vaex.from_arrays(x=x, y=y)
print(df)

display(df.x.values)
display(df.y.values)

It is useful to know that string operations in Vaex also work on lists of lists of strings (and also on lists of lists of lists of strings, and so on).

python

x = pa.array([['Reggie', 'Miller'], ['Michael', 'Jordan']])
df = vaex.from_arrays(x=x)
df.x.str.lower()