Arrow

DataFusion implements the Apache Arrow PyCapsule interface for importing and exporting DataFrames with zero copy. With this feature, any Python project that implements this interface can share data back and forth with DataFusion with zero copy.

We can demonstrate using pyarrow.

Importing to DataFusion

Here we will create an Arrow table and import it to DataFusion.

To import an Arrow table, use datafusion.context.SessionContext.from_arrow(). This will accept any Python object that implements __arrow_c_stream__ or __arrow_c_array__ and returns a StructArray. Common pyarrow sources you can use are:

• Array (but it must return a Struct Array)

• Record Batch

• Record Batch Reader

• Table

from datafusion import SessionContext
import pyarrow as pa

data = {"a": [1, 2, 3], "b": [4, 5, 6]}
table = pa.Table.from_pydict(data)

ctx = SessionContext()
df = ctx.from_arrow(table)
df

DataFrame()
+---+---+
| a | b |
+---+---+
| 1 | 4 |
| 2 | 5 |
| 3 | 6 |
+---+---+

Exporting from DataFusion

DataFusion DataFrames implement __arrow_c_stream__ PyCapsule interface, so any Python library that accepts these can import a DataFusion DataFrame directly.

Invoking __arrow_c_stream__ triggers execution of the underlying query, but batches are yielded incrementally rather than materialized all at once in memory. Consumers can process the stream as it arrives. The stream executes lazily, letting downstream readers pull batches on demand.

from datafusion import col, lit

df = df.select((col("a") * lit(1.5)).alias("c"), lit("df").alias("d"))
pa.table(df)

pyarrow.Table
c: double
d: string_view not null
----
c: [[1.5,3,4.5]]
d: [["df","df","df"]]