ContextQMD
Back to Arrow

MATLAB Interface to Apache Arrow

matlab/README.md

latest12.0 KB
Original Source
<!--- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -->

MATLAB Interface to Apache Arrow

Status

Warning The MATLAB interface is under active development and should be considered experimental.

This is a very early stage MATLAB interface to the Apache Arrow C++ libraries.

Currently, the MATLAB interface supports:

  1. Converting between a subset of Arrow Array types and MATLAB array types (see table below)
  2. Converting between MATLAB tables and arrow.tabular.RecordBatchs
  3. Creating Arrow Fields, Schemas, and Types
  4. Reading and writing Feather V1 files

Supported arrow.array.Array types are included in the table below.

NOTE: All Arrow Array classes listed below are part of the arrow.array package (e.g. arrow.array.Float64Array).

MATLAB Array TypeArrow Array Type
uint8UInt8Array
uint16UInt16Array
uint32UInt32Array
uint64UInt64Array
int8Int8Array
int16Int16Array
int32Int32Array
int64Int64Array
singleFloat32Array
doubleFloat64Array
logicalBooleanArray
stringStringArray
datetimeTimestampArray
datetimeDate32Array
datetimeDate64Array
durationTime32Array
durationTime64Array
cellListArray
tableStructArray

Prerequisites

To build the MATLAB Interface to Apache Arrow from source, the following software must be installed on the target machine:

  1. MATLAB
  2. CMake
  3. C++ compiler which supports C++20 (e.g. gcc on Linux, Xcode on macOS, or Visual Studio on Windows)
  4. Git

Setup

To set up a local working copy of the source code, start by cloning the apache/arrow GitHub repository using Git:

console
$ git clone https://github.com/apache/arrow.git

After cloning, change the working directory to the matlab subdirectory:

console
$ cd arrow/matlab

Build

To build the MATLAB interface, use CMake:

console
$ cmake -S . -B build
$ cmake --build build --config Release

Install

To install the MATLAB interface to the default software installation location for the target machine (e.g. /usr/local on Linux or C:\Program Files on Windows), pass the --target install flag to CMake.

console
$ cmake --build build --config Release --target install

As part of the install step, the installation directory is added to the MATLAB Search Path.

Note: This step may fail if the current user is lacking necessary filesystem permissions. If the install step fails, the installation directory can be manually added to the MATLAB Search Path using the addpath command.

Test

To run the MATLAB tests, start MATLAB in the arrow/matlab directory and call the runtests command on the test directory with IncludeSubFolders=true:

matlab
>> runtests("test", IncludeSubFolders=true);

Refer to Testing Guidelines for more information.

Usage

Included below are some example code snippets that illustrate how to use the MATLAB interface.

Arrow Array classes (i.e. arrow.array.<Array>)

Create an Arrow Float64Array from a MATLAB double array

matlab
>> matlabArray = double([1, 2, 3])

matlabArray =

     1     2     3

>> arrowArray = arrow.array(matlabArray)

arrowArray = 

  Float64Array with 3 elements and 0 null values:

    1 | 2 | 3

Create a MATLAB logical array from an Arrow BooleanArray

matlab
>> arrowArray = arrow.array([true, false, true])

arrowArray = 

  BooleanArray with 3 elements and 0 null values:

    true | false | true

>> matlabArray = toMATLAB(arrowArray)

matlabArray =

  3×1 logical array

   1
   0
   1

Specify Null Values when constructing an arrow.array.Int8Array

matlab
>> matlabArray = int8([122, -1, 456, -10, 789])

matlabArray =

  1×5 int8 row vector

    122     -1    127    -10    127

% Treat all negative array elements as Null
>> validElements = matlabArray > 0

validElements =

  1×5 logical array

   1   0   1   0   1

% Specify which values are Null/Valid by supplying a logical validity "mask"
>> arrowArray = arrow.array(matlabArray, Valid=validElements)

arrowArray = 

  Int8Array with 5 elements and 2 null values:

    122 | null | 127 | null | 127

Arrow RecordBatch class

Create an Arrow RecordBatch from a MATLAB table

matlab
>> matlabTable = table(["A"; "B"; "C"], [1; 2; 3], [true; false; true])

matlabTable =

  3x3 table

    Var1    Var2    Var3
    ____    ____    _____

    "A"      1      true
    "B"      2      false
    "C"      3      true

>> arrowRecordBatch = arrow.recordBatch(matlabTable)

arrowRecordBatch = 

  Arrow RecordBatch with 3 rows and 3 columns:

    Schema:

        Var1: String | Var2: Float64 | Var3: Boolean

    First Row:

        "A" | 1 | true

Create a MATLAB table from an Arrow RecordBatch

matlab
>> arrowRecordBatch

arrowRecordBatch = 

  Arrow RecordBatch with 3 rows and 3 columns:

    Schema:

        Var1: String | Var2: Float64 | Var3: Boolean

    First Row:

        "A" | 1 | true

>> matlabTable = table(arrowRecordBatch)

matlabTable =

  3x3 table

    Var1    Var2    Var3
    ____    ____    _____

    "A"      1      true
    "B"      2      false
    "C"      3      true

Create an Arrow RecordBatch from multiple Arrow Arrays

matlab
>> stringArray = arrow.array(["A", "B", "C"])

stringArray = 

  StringArray with 3 elements and 0 null values:

    "A" | "B" | "C"

>> timestampArray = arrow.array([datetime(1997, 01, 01), datetime(1998, 01, 01), datetime(1999, 01, 01)])

timestampArray = 

  TimestampArray with 3 elements and 0 null values:

    1997-01-01 00:00:00.000000 | 1998-01-01 00:00:00.000000 | 1999-01-01 00:00:00.000000

>> booleanArray = arrow.array([true, false, true])

booleanArray = 

  BooleanArray with 3 elements and 0 null values:

    true | false | true

>> arrowRecordBatch = arrow.tabular.RecordBatch.fromArrays(stringArray, timestampArray, booleanArray)

arrowRecordBatch = 

  Arrow RecordBatch with 3 rows and 3 columns:

    Schema:

        Column1: String | Column2: Timestamp | Column3: Boolean

    First Row:

        "A" | 1997-01-01 00:00:00.000000 | true

Extract a column from a RecordBatch by index

matlab
>> arrowRecordBatch = arrow.tabular.RecordBatch.fromArrays(stringArray, timestampArray, booleanArray)

arrowRecordBatch = 

  Arrow RecordBatch with 3 rows and 3 columns:

    Schema:

        Column1: String | Column2: Timestamp | Column3: Boolean

    First Row:

        "A" | 1997-01-01 00:00:00.000000 | true

>> timestampArray = arrowRecordBatch.column(2)

timestampArray = 

  TimestampArray with 3 elements and 0 null values:

    1997-01-01 00:00:00.000000 | 1998-01-01 00:00:00.000000 | 1999-01-01 00:00:00.000000

Arrow Type classes (i.e. arrow.type.<Type>)

Create an Arrow Int8Type object

matlab
>> type = arrow.int8()

type =

  Int8Type with properties:

    ID: Int8

Create an Arrow TimestampType object with a specific TimeUnit and TimeZone

matlab
>> type = arrow.timestamp(TimeUnit="Second", TimeZone="Asia/Kolkata")

type =

  TimestampType with properties:

          ID: Timestamp
    TimeUnit: Second
    TimeZone: "Asia/Kolkata"

Get the type enumeration ID for an Arrow Type object

matlab
>> type.ID

ans =

  ID enumeration

    Timestamp

>> type = arrow.string()

type =

  StringType with properties:

    ID: String

>> type.ID

ans =

  ID enumeration

    String

Arrow Field class

Create an Arrow Field with type Int8Type

matlab
>> field = arrow.field("Number", arrow.int8())

field = 

  Field with properties:

    Name: "Number"
    Type: [1x1 arrow.type.Int8Type]

>> field.Name

ans =

    "Number"

>> field.Type

ans =

  Int8Type with properties:

    ID: Int8

Create an Arrow Field with type StringType

matlab
>> field = arrow.field("Letter", arrow.string())

field = 

  Field with properties:

    Name: "Letter"
    Type: [1x1 arrow.type.StringType]

>> field.Name

ans =

    "Letter"

>> field.Type

ans =

  StringType with properties:

    ID: String

Extract an Arrow Field from an Arrow Schema by index

matlab
>> arrowSchema

arrowSchema = 

  Arrow Schema with 2 fields:

    Letter: String | Number: Int8

% Specify the field to extract by its index (i.e. 2)
>> field = arrowSchema.field(2)

field = 

  Field with properties:

    Name: "Number"
    Type: [1x1 arrow.type.Int8Type]

Extract an Arrow Field from an Arrow Schema by name

matlab
>> arrowSchema

arrowSchema = 

  Arrow Schema with 2 fields:

    Letter: String | Number: Int8

% Specify the field to extract by its name (i.e. "Letter")
>> field = arrowSchema.field("Letter")

field = 

  Field with properties:

    Name: "Letter"
    Type: [1x1 arrow.type.StringType]

Arrow Schema class

Create an Arrow Schema from multiple Arrow Fields

matlab
>> letter = arrow.field("Letter", arrow.string())

letter = 

  Field with properties:

    Name: "Letter"
    Type: [1x1 arrow.type.StringType]

>> number = arrow.field("Number", arrow.int8())

number = 

  Field with properties:

    Name: "Number"
    Type: [1x1 arrow.type.Int8Type]

>> schema = arrow.schema([letter, number])

schema = 

  Arrow Schema with 2 fields:

    Letter: String | Number: Int8

Get the Schema of an Arrow RecordBatch

matlab
>> matlabTable = table(["A"; "B"; "C"], [1; 2; 3], VariableNames=["Letter", "Number"])

matlabTable =

  3x2 table

    Letter    Number
    ______    ______

     "A"        1
     "B"        2
     "C"        3

>> arrowRecordBatch = arrow.recordBatch(matlabTable)

arrowRecordBatch = 

  Arrow RecordBatch with 3 rows and 2 columns:

    Schema:

        Letter: String | Number: Float64

    First Row:

        "A" | 1

>> arrowSchema = arrowRecordBatch.Schema

arrowSchema = 

  Arrow Schema with 2 fields:

    Letter: String | Number: Float64

Feather V1

Write a MATLAB table to a Feather V1 file

matlab
>> t = table(["A"; "B"; "C"], [1; 2; 3], [true; false; true])

t =

  3×3 table

    Var1    Var2    Var3
    ____    ____    _____

    "A"      1      true
    "B"      2      false
    "C"      3      true

>> filename = "table.feather";

>> featherwrite(filename, t)

Read a Feather V1 file into a MATLAB table

matlab
>> filename = "table.feather";

>> t = featherread(filename)

t =

  3×3 table

    Var1    Var2    Var3
    ____    ____    _____

    "A"      1      true
    "B"      2      false
    "C"      3      true