datafusion.plan#

This module supports physical and logical plans in DataFusion.

Classes#

`ExecutionPlan`	Represent nodes in the DataFusion Physical Plan.
`LogicalPlan`	Logical Plan.
`Metric`	A single execution metric with name, value, partition, and labels.
`MetricsSet`	A set of metrics for a single execution plan operator.

Module Contents#

class datafusion.plan.ExecutionPlan(plan: datafusion._internal.ExecutionPlan)#

Represent nodes in the DataFusion Physical Plan.

This constructor should not be called by the end user.

__repr__() → str#: Print a string representation of the physical plan.

children() → list[ExecutionPlan]#

Get a list of children ExecutionPlan that act as inputs to this plan.

The returned list will be empty for leaf nodes such as scans, will contain a single value for unary nodes, or two values for binary nodes (such as joins).

collect_metrics() → list[tuple[str, MetricsSet]]#

Return runtime statistics for each step of the query execution.

DataFusion executes a query as a pipeline of operators — for example a data source scan, followed by a filter, followed by a projection. After the DataFrame has been executed (via collect(), execute_stream(), etc.), each operator records statistics such as how many rows it produced and how much CPU time it consumed.

Each entry in the returned list corresponds to one operator that recorded metrics. The first element of the tuple is the operator’s description string — the same text shown by display_indent() — which identifies both the operator type and its key parameters, for example "FilterExec: column1@0 > 1" or "DataSourceExec: partitions=1".

Returns:: A list of (description, MetricsSet) tuples ordered from the outermost operator (top of the execution tree) down to the data-source leaves. Only operators that recorded at least one metric are included. Returns an empty list if called before the DataFrame has been executed.

display() → str#: Print the physical plan.

display_indent() → str#: Print an indented form of the physical plan.

static from_bytes(ctx: datafusion.context.SessionContext, data: bytes) → ExecutionPlan#

Create an ExecutionPlan from serialized protobuf bytes.

Decoding routes through the session’s installed PhysicalExtensionCodec. Tables created in memory from record batches are currently not supported.

static from_proto(ctx: datafusion.context.SessionContext, data: bytes) → ExecutionPlan#: Deprecated alias for from_bytes().

metrics() → MetricsSet | None#

Return metrics for this plan node, or None if this plan has no MetricsSet.

Some operators (e.g. DataSourceExec) eagerly initialize a MetricsSet when the plan is created, so this may return a set even before execution. Metric values (such as output_rows) are only meaningful after the DataFrame has been executed.

to_bytes(ctx: datafusion.context.SessionContext | None = None) → bytes#

Convert an ExecutionPlan into serialized protobuf bytes.

When ctx is supplied, encoding routes through the session’s installed PhysicalExtensionCodec. Tables created in memory from record batches are currently not supported.

to_proto() → bytes#: Deprecated alias for to_bytes().

_raw_plan#

property partition_count: int#: Returns the number of partitions in the physical plan.

class datafusion.plan.LogicalPlan(plan: datafusion._internal.LogicalPlan)#

Logical Plan.

A LogicalPlan is a node in a tree of relational operators (such as Projection or Filter).

Represents transforming an input relation (table) to an output relation (table) with a potentially different schema. Plans form a dataflow tree where data flows from leaves up to the root to produce the query result.

A LogicalPlan can be created by the SQL query planner, the DataFrame API, or programmatically (for example custom query languages).

This constructor should not be called by the end user.

__eq__(other: LogicalPlan) → bool#: Test equality.

__repr__() → str#: Generate a printable representation of the plan.

display() → str#: Print the logical plan.

display_graphviz() → str#

Print the graph visualization of the logical plan.

Returns a format`able structure that produces lines meant for graphical display using the `DOT language. This format can be visualized using software from [graphviz](https://graphviz.org/)

display_indent() → str#: Print an indented form of the logical plan.

display_indent_schema() → str#: Print an indented form of the schema for the logical plan.

static from_bytes(ctx: datafusion.context.SessionContext, data: bytes) → LogicalPlan#

Create a LogicalPlan from serialized protobuf bytes.

Decoding routes through the session’s installed LogicalExtensionCodec. Tables created in memory from record batches are currently not supported.

static from_proto(ctx: datafusion.context.SessionContext, data: bytes) → LogicalPlan#: Deprecated alias for from_bytes().

inputs() → list[LogicalPlan]#: Returns the list of inputs to the logical plan.

to_bytes(ctx: datafusion.context.SessionContext | None = None) → bytes#

Convert a LogicalPlan to serialized protobuf bytes.

When ctx is supplied, encoding routes through the session’s installed LogicalExtensionCodec so user FFI codecs (registered via SessionContext.with_logical_extension_codec()) see the encode path. With ctx=None a default codec is used. Tables created in memory from record batches are currently not supported.

to_proto() → bytes#: Deprecated alias for to_bytes().

to_variant() → Any#: Convert the logical plan into its specific variant.

_raw_plan#

class datafusion.plan.Metric(raw: datafusion._internal.Metric)#

A single execution metric with name, value, partition, and labels.

This constructor should not be called by the end user.

__repr__() → str#: Return a string representation of the metric.

labels() → dict[str, str]#

Return the labels associated with this metric.

Labels provide additional context for a metric. For example:

metric.labels()
# {'output_type': 'final'}

_raw#

property name: str#: The name of this metric (e.g. output_rows).

property partition: int | None#

The 0-based partition index this metric applies to.

Returns None for metrics that are not partition-specific (i.e. they apply globally across all partitions of the operator).

property value: int | datetime.datetime | None#

The value of this metric.

Returns an int for counters, gauges, and time-based metrics (nanoseconds), a datetime (UTC) for start_timestamp / end_timestamp metrics, or None when the value has not been set or is not representable.

property value_as_datetime: datetime.datetime | None#

The value as a UTC datetime for timestamp metrics.

Returns None for all non-timestamp metrics and for timestamp metrics whose value has not been set (e.g. before execution).

class datafusion.plan.MetricsSet(raw: datafusion._internal.MetricsSet)#

A set of metrics for a single execution plan operator.

A physical plan operator runs independently across one or more partitions. metrics() returns the raw per-partition Metric objects. The convenience properties (output_rows, elapsed_compute, etc.) automatically sum the named metric across all partitions, giving a single aggregate value for the operator as a whole.

This constructor should not be called by the end user.

__repr__() → str#: Return a string representation of the metrics set.

metrics() → list[Metric]#: Return all individual metrics in this set.

sum_by_name(name: str) → int | None#

Sum the named metric across all partitions.

Useful for accessing any metric not exposed as a first-class property. Returns None if no metric with the given name was recorded.

Parameters:: name – The metric name, e.g. "output_rows" or "elapsed_compute".

_raw#

property elapsed_compute: int | None#

Total CPU time (in nanoseconds) spent inside this operator’s execute loop.

Summed across all partitions. Returns None if no elapsed_compute metric was recorded.

property output_rows: int | None#: Sum of output_rows across all partitions.

property spill_count: int | None#

Number of times this operator spilled data to disk due to memory pressure.

This is a count of spill events, not a byte count. Summed across all partitions. Returns None if no spill_count metric was recorded.

property spilled_bytes: int | None#: Sum of spilled_bytes across all partitions.

property spilled_rows: int | None#: Sum of spilled_rows across all partitions.