Upgrade Guides#

DataFusion 56.0.0#

Note: DataFusion 56.0.0 has not been released yet. The information provided in this section pertains to features and changes that have already been merged to the main branch and are awaiting release in this version.

Session time zone now applies to mixed timestamp operations#

When datafusion.execution.time_zone is set, DataFusion now uses it to interpret timezone-naive timestamps in comparisons and subtraction with timezone-aware timestamps. This applies to binary comparisons, IN, BETWEEN, CASE expr WHEN, IN subqueries, and ANY or ALL comparisons.

Previously, mixed timestamp operations did not consistently use the configured session timezone. They could use the timezone-aware operand’s timezone or subtract raw timestamp values. The new behavior matches PostgreSQL and DuckDB.

The default remains None. Without a session timezone, comparisons and subtraction use the timezone-aware operand’s timezone. Applications that set this option need to review queries that mix timezone-aware and timezone-naive timestamps.

get_coerce_type_for_list and get_coerce_type_for_case_when are deprecated#

IN and CASE expr WHEN are defined in terms of =, and = now reads a timezone-naive timestamp in datafusion.execution.time_zone when it is compared against a timezone-aware one. These two functions have nowhere to take that setting from, so the planner no longer calls them and they now return a coercion that disagrees with the comparison operator they are defined in terms of.

Both remain in place and behave exactly as they did in 55.0.0. To follow the planner, fold comparison_coercion_with_session_timezone over the values yourself:

// 55.0.0
let coerced = get_coerce_type_for_list(&expr_type, &list_types);

// 56.0.0
let coerced = list_types.iter().try_fold(expr_type, |left, right| {
    comparison_coercion_with_session_timezone(&left, right, session_time_zone)
});

Pass options.execution.time_zone.as_ref().map(|tz| tz.as_str()) as session_time_zone, or None where no session is in scope. None reproduces the 55.0.0 behavior, except that NULL against a dictionary-encoded type coerces to the dictionary type rather than to its value type, which is what = has always done.

MSRV updated to 1.95.0#

The Minimum Supported Rust Version (MSRV) has been updated to 1.95.0.

Upgrade arrow/parquet to 60.0.0 and object_store to 0.14.2#

DataFusion 56.0.0 uses arrow and parquet 60.0.0, and object_store 0.14.2. This may require updates to your Cargo.toml if you have direct dependencies on these crates.

See the Arrow 60.0.0 release notes and the object_store 0.14.2 upgrade guide for details on breaking changes in those versions.

Field and schema metadata now use arrow’s Metadata type#

Arrow 60 introduced a dedicated Metadata type for field and schema metadata in place of HashMap<String, String>. Following that change, several DataFusion APIs now use Metadata as well, such as DFSchema::metadata and ExprSchema::metadata return &Metadata. CastExpr::target_metadata and TryCastExpr::target_metadata return Option<&Metadata>.

Migration guide:

// Before
let meta: &HashMap<String, String> = schema.metadata();
let field = Field::new("a", DataType::Int64, true)
    .with_metadata([("k".to_string(), "v".to_string())].into());

// After
let meta: &Metadata = schema.metadata();
let field = Field::new("a", DataType::Int64, true)
    .with_metadata(Metadata::new().with("k", "v"));

Metadata supports .get(), .iter(), .is_empty() and .extend() like HashMap, and converts to and from HashMap<String, String> and BTreeMap<String, String> via From, so most call sites need only a type change. Use FieldMetadata::to_hashmap where a HashMap is still required.

ParquetAccessPlan::into_overall_row_selection is deprecated#

ParquetAccessPlan::into_overall_row_selection is deprecated starting in 56.0.0. DataFusion now keeps selections local to each row group during Parquet scanning. The method remains available with its existing conversion and selection-length validation behavior, so direct callers continue to work but receive a deprecation warning.

Users who attach a ParquetAccessPlan to a PartitionedFile for DataFusion to read need no changes.

For custom readers, use ParquetAccessPlan::into_inner() to obtain the RowGroupAccess entries. Their positions are the original file row-group indexes, and each selection uses coordinates within its own row group. When migrating away from the deprecated method, callers must handle conversion and selection-length validation themselves. Parquet’s push decoder and async reader accept RowGroupSelection values through with_row_group_selections; the synchronous reader still requires with_row_groups and an overall with_row_selection.

RowGroupAccessPlanFilter::prune_by_range takes a RowGroupRangeAssignment#

The new datafusion.execution.parquet.row_group_range_assignment option decides which byte range of a split Parquet file reads each row group. The default, start_offset, keeps the previous behavior. midpoint matches Apache Spark.

RowGroupAccessPlanFilter::prune_by_range takes the assignment as a new argument. Pass RowGroupRangeAssignment::StartOffset for the previous behavior:

// Before
filter.prune_by_range(groups, &range);
// After
filter.prune_by_range(groups, &range, RowGroupRangeAssignment::StartOffset);

ParquetOptions and its generated protobuf struct gained a row_group_range_assignment field. Exhaustive struct literals must set it or use ..Default::default().

Missing Parquet null counts are treated as unknown#

DataFusion now preserves an omitted Parquet null_count statistic as unknown. Previously, it assumed zero nulls, which could discard matching NULL rows in IS NULL filters and ORDER BY ... NULLS FIRST LIMIT queries, or produce an incorrectly high COUNT(column) from metadata.

Queries over affected files may read or sort more data because these optimizations require a known null count:

  • Pruning row groups for IS NULL filters.

  • Computing COUNT(column) using only file metadata.

  • Pruning row groups with a dynamic NULLS FIRST TopK filter.

  • Eliminating a sort over nullable columns in sorted, non-overlapping files. Such queries may now retain a full SortExec, even when the data has no NULLs.

parquet-rs versions before 53.1.0 omitted zero null counts. This includes files produced with the older parquet-rs dependency used by DataFusion releases before 42.1.0. Arrow #6490 changed the writer to record known zero counts. Files with explicit counts retain their existing behavior.

To identify column chunks with bounds but no null count, run this query in the DataFusion CLI:

SELECT row_group_id, path_in_schema
FROM parquet_metadata('data.parquet')
WHERE (stats_min IS NOT NULL OR stats_max IS NOT NULL)
  AND stats_null_count IS NULL;

Rewriting affected files with a current DataFusion version, for example using COPY ... TO with Parquet statistics enabled, records the missing counts and restores optimizations that depend on them. Preserve the required data ordering and ordering metadata when rewriting sorted files.

Timestamp(u, Some(tz)) - Timestamp(u, None) reads the naive operand in tz#

Subtracting a timezone-naive timestamp from a timezone-aware one at the same time unit did not coerce at all in 55.0.0: the naive operand was read as UTC, which differed both from = on the same pair of values and from every other pairing of time units. It now coerces like the rest, so the naive operand is read in the aware operand’s time zone:

-- ts_tz is 2024-11-01T00:00:00-04:00 (America/New_York), ts is the naive
-- 2024-11-01T00:00:00; ts_tz = ts is true in both versions
-- 55.0.0: 0 days 4 hours 0 mins 0.000000000 secs
-- 56.0.0: 0 days 0 hours 0 mins 0.000000000 secs
SELECT ts_tz - ts FROM timestamps;

ForeignSession::create_physical_plan is unsupported#

ForeignSession::create_physical_plan no longer forwards to the library that owns the session. It now returns a NotImplemented error because forwarding can re-enter an installed foreign planner, and the execution-plan handle returned by the old callback cannot restore local Rust type identities for downcasting. The original FFI_SessionRef callback slot remains in place for ABI compatibility with DataFusion 55 consumers, but calling that callback returns the same error.

The session-owning library should instead export its original planner as a datafusion_ffi::query_planner::FFI_QueryPlanner before installing a foreign planner. The foreign planner can retain and invoke that handle to receive a serialized physical plan reconstructed with local type identities. See the datafusion_ffi::query_planner module documentation for the complete delegation pattern. ForeignSession::query_planner, optimize, and physical_optimizers continue to forward to the owning session across the FFI boundary.

Range partitioning supports retained samples and scaling#

RangePartitioning::new is deprecated. Prefer the validated RangePartitioning::try_new_with_samples(ordering, samples, partition_count). For exact boundaries, pass split_points.len() + 1 as the count, or keep using the validated try_new constructor. Retain additional samples when future scaling above that count is required; scaling does not create new sample values.

scale returns Option<RangePartitioning>. It returns None when the target partition count is zero or exceeds the retained sample capacity; callers can then retain the existing layout or choose another partitioning. Equality includes retained samples; use has_same_layout to compare the current key ordering and effective boundaries.

FFI_RangePartitioning has a new layout; rebuild FFI consumers against the compatible release. The generated PhysicalRangePartitioning protobuf struct also gains fields, affecting exhaustive struct literals. Older protobuf payloads remain readable, and effective split points remain available to older readers.

GroupColumn now requires values_preserving#

Custom implementations of the public GroupColumn trait must implement values_preserving. This method returns selected rows without changing the stored values or their group indices. It preserves the requested order and supports repeated indices.

Migration guide:

Implement values_preserving, call selection.validate_num_groups(self.len())? before reading, and return rows in selection.iter() order without changing the builder state.

datafusion-proto: common option and constraint conversions are fallible#

Protobuf conversions for CsvOptions, JsonOptions, ParquetCdcOptions, Constraint, and Constraints now reject integer values that do not fit usize. Their infallible From implementations have been replaced with TryFrom. The existing TryFrom conversions for ParquetOptions and TableParquetOptions now also validate all usize-backed fields. A constraint without a constraint_mode returns an error instead of panicking.

Migration guide:

// Before
let csv = CsvOptions::from(&proto_csv);
let cdc = ParquetCdcOptions::from(proto_cdc);
let constraints: Constraints = proto_constraints.into();

// After
let csv = CsvOptions::try_from(&proto_csv)?;
let cdc = ParquetCdcOptions::try_from(proto_cdc)?;
let constraints = Constraints::try_from(proto_constraints)?;

See issue #24170 for details.

ExecutionOptions has a new enable_nlj_coordinated_fallback field#

ExecutionOptions gained a public enable_nlj_coordinated_fallback: bool field (default true). It controls whether the memory-limited NestedLoopJoinExec fallback shares per-chunk build-side state across probe partitions, which is what lets LEFT, LEFT SEMI, LEFT ANTI, LEFT MARK and FULL joins spill instead of failing with ResourcesExhausted when the right side has several partitions.

Who is affected:

  • Users constructing ExecutionOptions (or ConfigOptions) with an exhaustive struct literal. Reading or SET-ing configuration is unaffected.

  • Distributed engines that run each output partition as an independent task. The coordination assumes all probe partitions run in one process; with one coordinator per task the shared probe-thread counter never reaches zero and the fallback would stall. Such engines should set the flag to false, which keeps the previous fail-fast behaviour for the affected join types.

Migration guide:

Set the new field, or fill it from Default:

// Before
ExecutionOptions {
    batch_size: 8192,
    // ... every other field ...
}

// After: set it explicitly
ExecutionOptions {
    batch_size: 8192,
    enable_nlj_coordinated_fallback: true,
    // ... every other field ...
}

// After: or let the remaining fields come from Default, which also keeps
// future field additions from breaking the literal
ExecutionOptions {
    batch_size: 8192,
    ..Default::default()
}

Distributed engines opting out:

let mut config = SessionConfig::new();
config.options_mut().execution.enable_nlj_coordinated_fallback = false;

ExecutionOptions::time_zone is now a validated ConfigTimeZone#

ExecutionOptions::time_zone changed from Option<String> to Option<ConfigTimeZone>. The value is now checked when it is set, so SET TIME ZONE = 'Asia/Taipei2' fails immediately instead of on a later query. The string is kept exactly as written (+08 stays +08).

Who is affected:

  • Code that reads or assigns ExecutionOptions::time_zone directly.

  • Code that sets the value through SQL SET or ConfigOptions::set needs no code changes, but invalid time zones are now rejected immediately.

Migration guide:

// Before
options.execution.time_zone = Some("America/Denver".to_string());
let tz: Option<String> = options.execution.time_zone.clone();

// After
options.execution.time_zone = Some("America/Denver".parse()?);
let tz: Option<&str> = options.execution.time_zone.as_ref().map(|tz| tz.as_str());

ExecutionOptions has a new soft_max_bytes_per_output_file field#

ExecutionOptions gained a public soft_max_bytes_per_output_file: ConfigNonZeroUsize field (default 4 GiB minus one byte, or 4,294,967,295 bytes). When writing multiple output files, DataFusion starts a new file once a writer reports that the encoded size has reached this target or the existing soft_max_rows_per_output_file target has been reached. The byte limit is soft: buffered batches, in-flight batch serialization, individual batch sizes, and file metadata can make a completed file larger than the target. It does not apply to single-file output.

Who is affected:

  • Users constructing ExecutionOptions (or ConfigOptions) with an exhaustive struct literal must provide the new field.

  • Users writing multiple output files can set a smaller byte target to rotate files before they reach the default size.

Migration guide:

Set the field explicitly, or fill it from Default as shown above. For example, to target 128 MiB output files in Rust:

let mut config = SessionConfig::new();
config.options_mut().execution.soft_max_bytes_per_output_file =
    ConfigNonZeroUsize::try_new(128 * 1024 * 1024)?;

The same setting can be changed with SQL:

SET datafusion.execution.soft_max_bytes_per_output_file = 134217728;

New field enable_rle_to_dictionary added to ParquetOptions#

ParquetOptions gained a new field enable_rle_to_dictionary: bool (default false). It controls whether top-level string and binary Parquet columns with dictionary pages are read as Dictionary<Int32, Utf8> / Dictionary<Int32, Binary> instead of their plain type.

ParquetOptions is a public struct without #[non_exhaustive], so downstream crates that construct it with an exhaustive struct literal will fail to compile with E0063: missing field 'enable_rle_to_dictionary'.

Migration guide:

Add enable_rle_to_dictionary: false to any exhaustive ParquetOptions { .. } literal, or use ..Default::default() to future-proof against further additions:

// Before
ParquetOptions {
    enable_page_index: true,
    // ... every other field ...
}

// After: set it explicitly
ParquetOptions {
    enable_page_index: true,
    enable_rle_to_dictionary: false,
    // ... every other field ...
}

// After: or let remaining fields come from Default
ParquetOptions {
    enable_page_index: true,
    ..Default::default()
}

datafusion.optimizer.use_statistics_registry is deprecated and ignored#

The datafusion.optimizer.use_statistics_registry config flag is deprecated and now ignored (setting it emits a deprecation warning). The pluggable StatisticsRegistry is always consulted during the single statistics walk: providers registered on the session take effect directly. With no providers registered the registry is a no-op, so default behavior is unchanged from the previous default (use_statistics_registry = false).

Who is affected:

  • Anyone who set datafusion.optimizer.use_statistics_registry. The setting is still accepted (with a deprecation warning) but has no effect, and will be removed in a future release.

Migration guide:

  • Drop any use_statistics_registry setting.

  • To use statistics providers, register them on the session with SessionStateBuilder::with_statistics_registry(...).

  • To opt out, register nothing (the default).

Generated protobuf JsonWriterOptions gained a compression_level field#

The generated public protobuf JsonWriterOptions struct now carries an optional compression_level so JSON sink plans preserve explicitly configured compression levels across serialization.

Who is affected:

  • Users constructing generated JsonWriterOptions values with an exhaustive struct literal.

Migration guide:

Set compression_level explicitly, or fill it from Default:

// Before
JsonWriterOptions { compression }

// After
JsonWriterOptions {
    compression,
    compression_level: None,
}
// or
JsonWriterOptions {
    compression,
    ..Default::default()
}

The protobuf wire format remains backward compatible.

See PR #24945 for details.

Generated protobuf Parquet structs gained state fields#

The generated public protobuf ParquetScanExecNode struct now carries an optional metadata_size_hint, and ParquetSink now carries optional sorting_columns. These fields preserve the corresponding Parquet source and sink settings across physical-plan serialization.

Who is affected:

  • Users constructing either generated struct with an exhaustive struct literal.

Migration guide:

Add metadata_size_hint: None to ParquetScanExecNode literals and sorting_columns: None to ParquetSink literals to retain the previous behavior, or use ..Default::default() for unspecified fields.

The protobuf wire format remains backward compatible.

See PR #25057 for details.

Generated protobuf CsvWriterOptions changed#

The generated CsvWriterOptions now includes compression_level, timestamp_tz_format, and terminator. Its four existing format fields changed from String to Option<String> to distinguish unset from explicitly empty formats. This affects exhaustive struct literals and direct format-field access.

Wrap existing formats in Some or use None, and initialize the new fields (explicitly or through Default):

// Before (unchanged fields omitted)
CsvWriterOptions {
    date_format: "%Y-%m-%d".to_string(),
    datetime_format: String::new(),
    timestamp_format: String::new(),
    time_format: String::new(),
    // ...
}

// After
CsvWriterOptions {
    date_format: Some("%Y-%m-%d".to_string()),
    datetime_format: None,
    timestamp_format: None,
    time_format: None,
    compression_level: None,
    timestamp_tz_format: None,
    terminator: Vec::new(),
    // ...
}

The protobuf wire format remains backward compatible.

See PR #25058 for details.

DefaultStatisticsProvider is deprecated#

datafusion_physical_plan::operator_statistics::DefaultStatisticsProvider is deprecated. It is redundant: the statistics walk (StatisticsContext) falls back to each operator’s statistics_from_inputs when the provider chain delegates or is empty, so a terminal “default” provider is no longer needed. It is no longer part of StatisticsRegistry::default_with_builtin_providers().

Migration guide:

  • Remove DefaultStatisticsProvider from any custom provider chain; register no terminal provider instead (the walk falls back on its own).

The bundled statistics providers are deprecated#

StatisticsRegistry::default_with_builtin_providers() and all the statistics providers bundled in datafusion_physical_plan::operator_statistics (FilterStatisticsProvider, ProjectionStatisticsProvider, PassthroughStatisticsProvider, AggregateStatisticsProvider, JoinStatisticsProvider, LimitStatisticsProvider and UnionStatisticsProvider) are deprecated. They duplicate or replace the estimation each operator already does in statistics_from_inputs, which is the default. StatisticsRegistry remains the extension point for user-defined providers.

Migration guide:

  • Stop registering the bundled providers; with no matching provider, the statistics walk uses the operator’s own estimate.

  • To keep a bundled estimation technique, implement it in a user-defined StatisticsProvider.

StatisticsRegistry::compute and compute_base are deprecated#

Use the walk instead:

StatisticsContext::new_with_registry(registry)
    .compute_extended(plan, &StatisticsArgs::new())?; // or .compute(...) for core Statistics

API change for floor and ceil UDF#

The output type of the floor and ceil UDFs has been changed from the exact input type to a rescaled type with the same bit width. For example, for input Decimal32(7,2) floor now returns Decimal32(6,0), where the new precision is p - s + 1, matching Spark’s behaviour. See [#24703] for more details.

Who is affected:

  • Users storing query result with these UDFs in a fixed schema

  • Users relying on arrow_typeof for these UDFs

Migration guide:

Change the expected type or wrap the expression in CAST. It’s recommended to avoid relying on decimal’s exact precision and scale.

map_extract / element_at return an empty list for absent keys#

map_extract (and its alias element_at) previously returned a single-element list containing NULL when the key was not present in the map. It now returns an empty list, matching the documented behavior and DuckDB. Two related cases changed at the same time, also matching DuckDB:

  • A NULL map input now yields NULL instead of [NULL].

  • A NULL lookup key now yields [] instead of [NULL].

A key that is present with a NULL value still returns [NULL], so absent keys and NULL values are now distinguishable.

Migration guide:

-- Before
SELECT map_extract(MAP {'a': 1}, 'missing');  -- [NULL]

-- After
SELECT map_extract(MAP {'a': 1}, 'missing');  -- []

Expressions that assumed the result always has exactly one element, for example by checking its length, unnesting it, or comparing it to [NULL], should treat an empty list as the absent-key case instead.

See issue #24981 and issue #24983 for details.

DataFrame::from_columns accepts IntoIterator#

DataFrame::from_columns now accepts any IntoIterator<Item = (&str, ArrayRef)> instead of specifically accepting a Vec<(&str, ArrayRef)>.

// Existing Vec usage continues to work
let df = DataFrame::from_columns(vec![
    ("id", id),
    ("name", name),
])?;

// Arrays can now be used directly
let df = DataFrame::from_columns([
    ("id", id),
    ("name", name),
])?;

Most existing call sites using Vec require no changes. Code that relies on the exact non-generic function signature of DataFrame::from_columns may need to be updated to account for the new generic API.

Aggregate ordering state field names are now namespaced#

Ordering expressions exposed as top-level aggregate state fields now use names derived from the aggregate name and their ordering position.

For example:

first_value(value)[ordering_0]

is now used instead of an unqualified ordering field name such as:

timestamp@0

This guarantees unique aggregate state field names and allows DFSchema::check_names() to be enforced.

Who is affected:

Users or integrations that inspect aggregate state field names directly, including custom UDAFs and FFI integrations.

Physical filter pushdown resolves columns by position#

datafusion_physical_plan::filter_pushdown::ChildFilterDescription::from_child and FilterDescription::from_children now resolve filter columns by position instead of looking them up by name. This prevents incorrect results when a child schema contains duplicate column names. from_child requires the child field at each referenced position to have the same name as the filter column.

ChildFilterDescription::from_child_with_allowed_indices is deprecated but preserves its previous name-based mapping to the first matching child field. Migrate to from_child_with_column_mapping because name resolution is ambiguous when the child schema contains duplicate field names.

Migration guide:

Use from_child (or from_children for multiple children) when the parent and child schemas have matching column positions and names. When a node projects, reorders, or renames columns, use from_child_with_column_mapping with an explicit map from parent output indices to child input indices. Columns absent from the mapping cannot be pushed down.

For example, if the parent outputs [a, b] and the child outputs [b, a], a filter on a@0 must map to child column a@1:

use std::collections::{HashMap, HashSet};
use datafusion_physical_plan::filter_pushdown::ChildFilterDescription;

// Before: allow parent column 0 and resolve "a" by name in the child.
let description = ChildFilterDescription::from_child_with_allowed_indices(
    &parent_filters,
    HashSet::from([0]),
    &child,
)?;

// After: explicitly map parent column 0 to child column 1.
let description = ChildFilterDescription::from_child_with_column_mapping(
    &parent_filters,
    HashMap::from([(0, 1)]),
    &child,
)?;

Default datafusion.sql_parser.recursion_limit raised from 50 to 51#

DataFusion now uses sqlparser 0.63.0, which adds recursion guards to more parse functions (for example, data type and INTERVAL parsing). As a result, the same SQL statement consumes slightly more of the parser recursion budget than it did with previous sqlparser versions. In particular, every expression now needs one additional level of depth.

To avoid rejecting queries that previously parsed successfully, the default value of the datafusion.sql_parser.recursion_limit configuration option, and of DFParserBuilder::with_recursion_limit, has been raised from 50 to 51.

If you set datafusion.sql_parser.recursion_limit explicitly to a value close to the depth of your queries, you may see RecursionLimitExceeded errors after upgrading and should raise the limit accordingly.

See PR #25278 for details.

ensure_distribution is deprecated in favour of ensure_distribution_with_stats#

datafusion_physical_optimizer::enforce_distribution::ensure_distribution now takes its StatisticsContext from the caller, so one context (and its memoization cache) can be shared across a whole bottom-up traversal. The old two-argument form is kept as a deprecated wrapper that allocates a fresh context per call, which recomputes each shared subtree’s statistics once per ancestor.

// Before
let plan = DistributionContext::new_default(plan)
    .transform_up(|ctx| ensure_distribution(ctx, config))
    .data()?;

// After: one context for the whole walk. `StatsCache` is keyed by raw plan-node
// pointers, so reset it whenever a node's plan pointer actually changed.
let stats_ctx = StatisticsContext::new();
let plan = DistributionContext::new_default(plan)
    .transform_up(|ctx| {
        let before = Arc::clone(&ctx.plan);
        let result = ensure_distribution_with_stats(ctx, config, &stats_ctx)?;
        if !Arc::ptr_eq(&before, &result.data.plan) {
            stats_ctx.reset_cache();
        }
        Ok(result)
    })
    .data()?;

Callers that only need the previous behaviour can keep using the deprecated form, or pass a freshly constructed StatisticsContext per call.

MergeIntoOp requires the SQL-visible target qualifier#

MergeIntoOp now stores the SQL-visible target qualifier separately from the target provider identity. This prevents an aliased target from being confused with a source relation that uses the target table’s name.

The struct is now non-exhaustive. Replace 55.0 struct literals with MergeIntoOp::new:

// DataFusion 55.0
let op = MergeIntoOp { on, clauses };

// DataFusion 56.0
let op = MergeIntoOp::new(target_qualifier, on, clauses);

target_qualifier must be the relation name referenced by the MERGE expressions: the target alias when one is present, otherwise the target table reference. DmlStatement::table_name remains the provider/catalog identity.

MergeIntoOpNode protobuf literals require target_qualifier#

The generated MergeIntoOpNode type has a new optional field. Code constructing the generated type with a struct literal must provide it:

// DataFusion 55.0
let node = protobuf::MergeIntoOpNode { on, clauses };

// DataFusion 56.0
let node = protobuf::MergeIntoOpNode {
    on,
    clauses,
    target_qualifier: Some(protobuf::TableReference::from(target_qualifier)),
};

Wire compatibility is directional:

  • A 56.0 reader accepts a 55.0 payload. When target_qualifier is absent, it falls back to DmlNode.table_name, matching the 55.0 representation.

  • A 55.0 reader must not consume an alias-preserving 56.0 MERGE payload. It ignores the unknown field but cannot preserve the qualifier required by the expressions, which can cause resolution failure or incorrect rebinding.

FunctionalDependence has a new null_equality field#

FunctionalDependence now records the NULL semantics its dependency holds under. Build it with FunctionalDependence::new(..) and, when needed, .with_null_equality(NullEquality::NullEqualsNull) instead of a struct literal.

Migration guide:

// Before
let dep = FunctionalDependence {
    source_indices,
    target_indices,
    nullable,
    mode: Dependency::Single,
};

// After
let dep = FunctionalDependence::new(source_indices, target_indices, nullable)
    .with_mode(Dependency::Single)
    .with_null_equality(NullEquality::NullEqualsNull);

FileSource::exact_filter controls scan equivalence properties#

FileScanConfig no longer derives equivalence properties (constant columns and equal columns) from FileSource::filter. It uses the new FileSource::exact_filter method instead. This method returns the part of the filter that every output row satisfies. A filter that the source uses only for pruning (for example, Parquet row group statistics when datafusion.execution.parquet.pushdown_filters is false) lets non-matching rows through, and equivalences derived from it gave wrong query results.

exact_filter returns None by default. ParquetSource returns the pushed-down conjuncts when pushdown_filters is enabled.

Migration guide:

If your custom FileSource applies its filter to every row, implement exact_filter to keep the equivalence properties of the scan:

impl FileSource for MySource {
    fn exact_filter(&self) -> Option<Arc<dyn PhysicalExpr>> {
        self.filter()
    }
    // ...
}

TableProvider::scan_with_args supports skip pushdown#

The ScanArgs struct now carries a new skip (number of rows to skip) field alongside limit. TableProvider::scan_with_args() by default forwards to scan(), so existing TableProvider implementations keep compiling and working unchanged; implementations that want to take advantage of skip (or future ScanArgs fields) can override scan_with_args in addition to scan.

A new TableProvider::supports_skip_pushdown() -> bool (default false) lets a provider guarantee it honors ScanArgs::skip exactly — that a scan built with skip omits exactly the first skip rows it would otherwise have produced, not just approximately. ListingTable overrides this to true and honors skip.

DataFusion’s SQL/DataFrame optimizer (push_down_limit) now populates LogicalPlan::TableScan::skip from a query’s LIMIT ... OFFSET ... clause, but only when the scan’s source reports supports_skip_pushdown() == true. In that case the skip is moved entirely into the scan and the remaining Limit above it no longer re-applies it; every other provider keeps today’s behavior unchanged, with the skip always enforced above the scan. skip is also reachable directly, independent of the optimizer, by constructing a TableScan (for example via LogicalPlanBuilder::scan_with_filters_fetch_skip) or by calling scan_with_args with ScanArgs::with_skip.

This pushdown is also preserved end-to-end through plan serialization: datafusion-substrait’s producer folds a scan’s skip into the FetchRel it emits for the wrapping Limit (Substrait’s ReadRel has no field of its own for it), and datafusion-proto’s ListingTableScanNode, ViewTableScanNode, and CustomTableScanNode messages gained optional fetch/skip fields so a TableScan’s own fetch/skip — previously silently dropped — now round-trips exactly.

Who is affected#

  • Custom TableProvider implementations that want skip pushdown: override scan_with_args to read ScanArgs::skip() and honor it exactly, and override supports_skip_pushdown to return true. Once you do, LIMIT ... OFFSET ... queries against that provider will have their skip pushed into the scan automatically.

  • Callers that already build a scan through ScanArgs/scan_with_args (for example custom query planners) can now also set skip.

API breaks#

The public TableScan struct has one new public field - skip and one field whose type is changed - statistics_requests. The addition of skip field increases the size of TableScan and thus the size of LogicalPlan enum. To keep the size of LogicalPlan reasonable the type of the TableScan::statistics_requests field is changed from BTreeSet<StatisticsRequest> (24 bytes) to Box<BTreeSet<StatisticsRequest>> (8 bytes). The users’ applications are recommended to use TableScanBuilder to construct new instances of TableScan instead of instantiating directly TableScan { ... } to avoid the API breaks.

Example#

let plan = provider
    .scan_with_args(
        &state,
        ScanArgs::default()
            .with_projection(projection)
            .with_filters(Some(&filters))
            .with_limit(limit)
            .with_skip(skip),
    )
    .await?
    .into_inner();
On this page
Edit on GitHub
Show Source