datafusion.substrait¶

This module provides support for using substrait with datafusion.

For additional information about substrait, see https://substrait.io/ for more information about substrait.

Classes¶

`Consumer`	Generates a logical plan from a substrait plan.
`Plan`	A class representing an encodable substrait plan.
`Producer`	Generates substrait plans from a logical plan.
`Serde`	Provides the `Substrait` serialization and deserialization.

Module Contents¶

class datafusion.substrait.Consumer¶

Generates a logical plan from a substrait plan.

static from_substrait_plan(ctx: datafusion.context.SessionContext, plan: Plan) → datafusion._internal.LogicalPlan¶

Convert a Substrait plan to a DataFusion LogicalPlan.

Parameters:

ctx – SessionContext to use.
plan – Substrait plan to convert.

Returns:

LogicalPlan.

class datafusion.substrait.Plan(plan: datafusion._internal.substrait.Plan)¶

A class representing an encodable substrait plan.

Create a substrait plan.

The user should not have to call this constructor directly. Rather, it should be created via Serde or py:class:Producer classes in this module.

encode() → bytes¶

Encode the plan to bytes.

Returns:: Encoded plan.

plan_internal¶

class datafusion.substrait.Producer¶

Generates substrait plans from a logical plan.

static to_substrait_plan(logical_plan: datafusion._internal.LogicalPlan, ctx: datafusion.context.SessionContext) → Plan¶

Convert a DataFusion LogicalPlan to a Substrait plan.

Parameters:

logical_plan – LogicalPlan to convert.
ctx – SessionContext to use.

Returns:

Substrait plan.

class datafusion.substrait.Serde¶

Provides the Substrait serialization and deserialization.

static deserialize(path: str | pathlib.Path) → Plan¶

Deserialize a Substrait plan from a file.

Parameters:: path – Path to read the Substrait plan from.
Returns:: Substrait plan.

static deserialize_bytes(proto_bytes: bytes) → Plan¶

Deserialize a Substrait plan from bytes.

Parameters:: proto_bytes – Bytes to read the Substrait plan from.
Returns:: Substrait plan.

static serialize(sql: str, ctx: datafusion.context.SessionContext, path: str | pathlib.Path) → None¶

Serialize a SQL query to a Substrait plan and write it to a file.

Parameters:

sql – SQL query to serialize.
ctx – SessionContext to use.
path – Path to write the Substrait plan to.

static serialize_bytes(sql: str, ctx: datafusion.context.SessionContext) → bytes¶

Serialize a SQL query to a Substrait plan as bytes.

Parameters:

sql – SQL query to serialize.
ctx – SessionContext to use.

Returns:

Substrait plan as bytes.

static serialize_to_plan(sql: str, ctx: datafusion.context.SessionContext) → Plan¶

Serialize a SQL query to a Substrait plan.

Args: sql: SQL query to serialize. ctx: SessionContext to use.

Returns:: Substrait plan.

datafusion.record_batch

datafusion.udf