datafusion¶

DataFusion python package.

This is a Python library that binds to Apache Arrow in-memory query engine DataFusion. See https://datafusion.apache.org/python for more information.

Submodules¶

Attributes¶

DFSchema

Classes¶

`Accumulator`	Defines how an `AggregateUDF` accumulates values.
`AggregateUDF`	Class for performing scalar user-defined functions (UDF).
`Catalog`	DataFusion data catalog.
`Database`	DataFusion Database.
`ExecutionPlan`	Represent nodes in the DataFusion Physical Plan.
`Expr`	Expression object.
`LogicalPlan`	Logical Plan.
`RecordBatch`	This class is essentially a wrapper for `pyarrow.RecordBatch`.
`RecordBatchStream`	This class represents a stream of record batches.
`RuntimeConfig`	Runtime configuration options.
`SQLOptions`	Options to be used when performing SQL queries.
`ScalarUDF`	Class for performing scalar user-defined functions (UDF).
`SessionConfig`	Session configuration options.
`Table`	DataFusion table.
`WindowFrame`	Defines a window frame for performing window operations.
`WindowUDF`	Class for performing window user-defined functions (UDF).

Functions¶

`col`(value)	Create a column expression.
`column`(value)	Create a column expression.
`lit`(value)	Create a literal expression.
`literal`(value)	Create a literal expression.

Package Contents¶

class datafusion.Accumulator¶

Defines how an AggregateUDF accumulates values.

abstract evaluate() → pyarrow.Scalar¶: Return the resultant value.

abstract merge(states: List[pyarrow.Array]) → None¶: Merge a set of states.

abstract state() → List[pyarrow.Scalar]¶: Return the current state.

abstract update(*values: pyarrow.Array) → None¶: Evaluate an array of values and update state.

class datafusion.AggregateUDF(name: str | None, accumulator: Callable[[], Accumulator], input_types: list[pyarrow.DataType], return_type: pyarrow.DataType, state_type: list[pyarrow.DataType], volatility: Volatility | str)¶

Class for performing scalar user-defined functions (UDF).

Aggregate UDFs operate on a group of rows and return a single value. See also ScalarUDF for operating on a row by row basis.

Instantiate a user-defined aggregate function (UDAF).

See udaf() for a convenience function and argument descriptions.

__call__(*args: datafusion.expr.Expr) → datafusion.expr.Expr¶

Execute the UDAF.

This function is not typically called by an end user. These calls will occur during the evaluation of the dataframe.

static udaf(accum: Callable[[], Accumulator], input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, state_type: list[pyarrow.DataType], volatility: Volatility | str, name: str | None = None) → AggregateUDF¶

Create a new User-Defined Aggregate Function.

If your Accumulator can be instantiated with no arguments, you can simply pass it’s type as accum. If you need to pass additional arguments to it’s constructor, you can define a lambda or a factory method. During runtime the Accumulator will be constructed for every instance in which this UDAF is used. The following examples are all valid.

import pyarrow as pa
import pyarrow.compute as pc

class Summarize(Accumulator):
    def __init__(self, bias: float = 0.0):
        self._sum = pa.scalar(bias)

    def state(self) -> List[pa.Scalar]:
        return [self._sum]

    def update(self, values: pa.Array) -> None:
        self._sum = pa.scalar(self._sum.as_py() + pc.sum(values).as_py())

    def merge(self, states: List[pa.Array]) -> None:
        self._sum = pa.scalar(self._sum.as_py() + pc.sum(states[0]).as_py())

    def evaluate(self) -> pa.Scalar:
        return self._sum

def sum_bias_10() -> Summarize:
    return Summarize(10.0)

udaf1 = udaf(Summarize, pa.float64(), pa.float64(), [pa.float64()], "immutable")
udaf2 = udaf(sum_bias_10, pa.float64(), pa.float64(), [pa.float64()], "immutable")
udaf3 = udaf(lambda: Summarize(20.0), pa.float64(), pa.float64(), [pa.float64()], "immutable")

Parameters:

accum – The accumulator python function.
input_types – The data types of the arguments to accum.
return_type – The data type of the return value.
state_type – The data types of the intermediate accumulation.
volatility – See Volatility for allowed values.
name – A descriptive name for the function.

Returns:

A user-defined aggregate function, which can be used in either data aggregation or window function calls.

_udaf¶

class datafusion.Catalog(catalog: datafusion._internal.Catalog)¶

DataFusion data catalog.

This constructor is not typically called by the end user.

database(name: str = 'public') → Database¶: Returns the database with the given name from this catalog.

names() → list[str]¶: Returns the list of databases in this catalog.

catalog¶

class datafusion.Database(db: datafusion._internal.Database)¶

DataFusion Database.

This constructor is not typically called by the end user.

names() → set[str]¶: Returns the list of all tables in this database.

table(name: str) → Table¶: Return the table with the given name from this database.

db¶

class datafusion.ExecutionPlan(plan: datafusion._internal.ExecutionPlan)¶

Represent nodes in the DataFusion Physical Plan.

This constructor should not be called by the end user.

__repr__() → str¶: Print a string representation of the physical plan.

children() → List[ExecutionPlan]¶

Get a list of children ExecutionPlan that act as inputs to this plan.

The returned list will be empty for leaf nodes such as scans, will contain a single value for unary nodes, or two values for binary nodes (such as joins).

display() → str¶: Print the physical plan.

display_indent() → str¶: Print an indented form of the physical plan.

static from_proto(ctx: datafusion.context.SessionContext, data: bytes) → ExecutionPlan¶

Create an ExecutionPlan from protobuf bytes.

Tables created in memory from record batches are currently not supported.

to_proto() → bytes¶

Convert an ExecutionPlan into protobuf bytes.

Tables created in memory from record batches are currently not supported.

_raw_plan¶

property partition_count: int¶: Returns the number of partitions in the physical plan.

class datafusion.Expr(expr: datafusion._internal.expr.Expr)¶

Expression object.

Expressions are one of the core concepts in DataFusion. See Expressions in the online documentation for more information.

This constructor should not be called by the end user.

__add__(rhs: Any) → Expr¶

Addition operator.