datafusion.substrait

This module provides support for using substrait with datafusion.

For additional information about substrait, see https://substrait.io/ for more information about substrait.

Classes

Consumer

Generates a logical plan from a substrait plan.

Plan

A class representing an encodable substrait plan.

Producer

Generates substrait plans from a logical plan.

Serde

Provides the Substrait serialization and deserialization.

Module Contents

class datafusion.substrait.Consumer

Generates a logical plan from a substrait plan.

static from_substrait_plan(ctx: datafusion.context.SessionContext, plan: Plan) datafusion._internal.LogicalPlan

Convert a Substrait plan to a DataFusion LogicalPlan.

Parameters:
  • ctx – SessionContext to use.

  • plan – Substrait plan to convert.

Returns:

LogicalPlan.

class datafusion.substrait.Plan(plan: datafusion._internal.substrait.Plan)

A class representing an encodable substrait plan.

Create a substrait plan.

The user should not have to call this constructor directly. Rather, it should be created via Serde or py:class:Producer classes in this module.

encode() bytes

Encode the plan to bytes.

Returns:

Encoded plan.

plan_internal
class datafusion.substrait.Producer

Generates substrait plans from a logical plan.

static to_substrait_plan(logical_plan: datafusion._internal.LogicalPlan, ctx: datafusion.context.SessionContext) Plan

Convert a DataFusion LogicalPlan to a Substrait plan.

Parameters:
  • logical_plan – LogicalPlan to convert.

  • ctx – SessionContext to use.

Returns:

Substrait plan.

class datafusion.substrait.Serde

Provides the Substrait serialization and deserialization.

static deserialize(path: str | pathlib.Path) Plan

Deserialize a Substrait plan from a file.

Parameters:

path – Path to read the Substrait plan from.

Returns:

Substrait plan.

static deserialize_bytes(proto_bytes: bytes) Plan

Deserialize a Substrait plan from bytes.

Parameters:

proto_bytes – Bytes to read the Substrait plan from.

Returns:

Substrait plan.

static serialize(sql: str, ctx: datafusion.context.SessionContext, path: str | pathlib.Path) None

Serialize a SQL query to a Substrait plan and write it to a file.

Parameters:
  • sql – SQL query to serialize.

  • ctx – SessionContext to use.

  • path – Path to write the Substrait plan to.

static serialize_bytes(sql: str, ctx: datafusion.context.SessionContext) bytes

Serialize a SQL query to a Substrait plan as bytes.

Parameters:
  • sql – SQL query to serialize.

  • ctx – SessionContext to use.

Returns:

Substrait plan as bytes.

static serialize_to_plan(sql: str, ctx: datafusion.context.SessionContext) Plan

Serialize a SQL query to a Substrait plan.

Args: sql: SQL query to serialize. ctx: SessionContext to use.

Returns:

Substrait plan.