Arrow¶

DataFusion implements the Apache Arrow PyCapsule interface for importing and exporting DataFrames with zero copy. With this feature, any Python project that implements this interface can share data back and forth with DataFusion with zero copy.

We can demonstrate using pyarrow.

Importing to DataFusion¶

Here we will create an Arrow table and import it to DataFusion.

To import an Arrow table, use datafusion.context.SessionContext.from_arrow(). This will accept any Python object that implements __arrow_c_stream__ or __arrow_c_array__ and returns a StructArray. Common pyarrow sources you can use are:

Array (but it must return a Struct Array)
Record Batch
Record Batch Reader
Table

In [1]: from datafusion import SessionContext

In [2]: import pyarrow as pa

In [3]: data = {"a": [1, 2, 3], "b": [4, 5, 6]}

In [4]: table = pa.Table.from_pydict(data)

In [5]: ctx = SessionContext()

In [6]: df = ctx.from_arrow(table)

In [7]: df
Out[7]: 
DataFrame()
+---+---+
| a | b |
+---+---+
| 1 | 4 |
| 2 | 5 |
| 3 | 6 |
+---+---+

Exporting from DataFusion¶

DataFusion DataFrames implement __arrow_c_stream__ PyCapsule interface, so any Python library that accepts these can import a DataFusion DataFrame directly.

Invoking __arrow_c_stream__ triggers execution of the underlying query, but batches are yielded incrementally rather than materialized all at once in memory. Consumers can process the stream as it arrives. The stream executes lazily, letting downstream readers pull batches on demand.

In [8]: from datafusion import col, lit

In [9]: df = df.select((col("a") * lit(1.5)).alias("c"), lit("df").alias("d"))

In [10]: pa.table(df)
Out[10]: 
pyarrow.Table
c: double
d: string_view not null
----
c: [[1.5,3,4.5]]
d: [["df","df","df"]]

Avro