Upgrade Guides#

DataFusion 56.0.0#

Note: DataFusion 56.0.0 has not been released yet. The information provided in this section pertains to features and changes that have already been merged to the main branch and are awaiting release in this version.

Missing Parquet null counts are treated as unknown#

DataFusion now preserves an omitted Parquet null_count statistic as unknown. Previously, it assumed zero nulls, which could discard matching NULL rows in IS NULL filters and ORDER BY ... NULLS FIRST LIMIT queries, or produce an incorrectly high COUNT(column) from metadata.

Queries over affected files may read or sort more data because these optimizations require a known null count:

  • Pruning row groups for IS NULL filters.

  • Computing COUNT(column) using only file metadata.

  • Pruning row groups with a dynamic NULLS FIRST TopK filter.

  • Eliminating a sort over nullable columns in sorted, non-overlapping files. Such queries may now retain a full SortExec, even when the data has no NULLs.

parquet-rs versions before 53.1.0 omitted zero null counts. This includes files produced with the older parquet-rs dependency used by DataFusion releases before 42.1.0. Arrow #6490 changed the writer to record known zero counts. Files with explicit counts retain their existing behavior.

To identify column chunks with bounds but no null count, run this query in the DataFusion CLI:

SELECT row_group_id, path_in_schema
FROM parquet_metadata('data.parquet')
WHERE (stats_min IS NOT NULL OR stats_max IS NOT NULL)
  AND stats_null_count IS NULL;

Rewriting affected files with a current DataFusion version, for example using COPY ... TO with Parquet statistics enabled, records the missing counts and restores optimizations that depend on them. Preserve the required data ordering and ordering metadata when rewriting sorted files.

ForeignSession::create_physical_plan is unsupported#

ForeignSession::create_physical_plan no longer forwards to the library that owns the session. It now returns a NotImplemented error because forwarding can re-enter an installed foreign planner, and the execution-plan handle returned by the old callback cannot restore local Rust type identities for downcasting. The original FFI_SessionRef callback slot remains in place for ABI compatibility with DataFusion 55 consumers, but calling that callback returns the same error.

The session-owning library should instead export its original planner as a datafusion_ffi::query_planner::FFI_QueryPlanner before installing a foreign planner. The foreign planner can retain and invoke that handle to receive a serialized physical plan reconstructed with local type identities. See the datafusion_ffi::query_planner module documentation for the complete delegation pattern. ForeignSession::query_planner, optimize, and physical_optimizers continue to forward to the owning session across the FFI boundary.

GroupColumn now requires values_preserving#

Custom implementations of the public GroupColumn trait must implement values_preserving. This method returns selected rows without changing the stored values or their group indices. It preserves the requested order and supports repeated indices.

Migration guide:

Implement values_preserving, call selection.validate_num_groups(self.len())? before reading, and return rows in selection.iter() order without changing the builder state.

datafusion-proto: common option and constraint conversions are fallible#

Protobuf conversions for CsvOptions, JsonOptions, ParquetCdcOptions, Constraint, and Constraints now reject integer values that do not fit usize. Their infallible From implementations have been replaced with TryFrom. The existing TryFrom conversions for ParquetOptions and TableParquetOptions now also validate all usize-backed fields. A constraint without a constraint_mode returns an error instead of panicking.

Migration guide:

// Before
let csv = CsvOptions::from(&proto_csv);
let cdc = ParquetCdcOptions::from(proto_cdc);
let constraints: Constraints = proto_constraints.into();

// After
let csv = CsvOptions::try_from(&proto_csv)?;
let cdc = ParquetCdcOptions::try_from(proto_cdc)?;
let constraints = Constraints::try_from(proto_constraints)?;

See issue #24170 for details.

ExecutionOptions has a new enable_nlj_coordinated_fallback field#

ExecutionOptions gained a public enable_nlj_coordinated_fallback: bool field (default true). It controls whether the memory-limited NestedLoopJoinExec fallback shares per-chunk build-side state across probe partitions, which is what lets LEFT, LEFT SEMI, LEFT ANTI, LEFT MARK and FULL joins spill instead of failing with ResourcesExhausted when the right side has several partitions.

Who is affected:

  • Users constructing ExecutionOptions (or ConfigOptions) with an exhaustive struct literal. Reading or SET-ing configuration is unaffected.

  • Distributed engines that run each output partition as an independent task. The coordination assumes all probe partitions run in one process; with one coordinator per task the shared probe-thread counter never reaches zero and the fallback would stall. Such engines should set the flag to false, which keeps the previous fail-fast behaviour for the affected join types.

Migration guide:

Set the new field, or fill it from Default:

// Before
ExecutionOptions {
    batch_size: 8192,
    // ... every other field ...
}

// After: set it explicitly
ExecutionOptions {
    batch_size: 8192,
    enable_nlj_coordinated_fallback: true,
    // ... every other field ...
}

// After: or let the remaining fields come from Default, which also keeps
// future field additions from breaking the literal
ExecutionOptions {
    batch_size: 8192,
    ..Default::default()
}

Distributed engines opting out:

let mut config = SessionConfig::new();
config.options_mut().execution.enable_nlj_coordinated_fallback = false;

datafusion.optimizer.use_statistics_registry is deprecated and ignored#

The datafusion.optimizer.use_statistics_registry config flag is deprecated and now ignored (setting it emits a deprecation warning). The pluggable StatisticsRegistry is always consulted during the single statistics walk: providers registered on the session take effect directly. With no providers registered the registry is a no-op, so default behavior is unchanged from the previous default (use_statistics_registry = false).

Who is affected:

  • Anyone who set datafusion.optimizer.use_statistics_registry. The setting is still accepted (with a deprecation warning) but has no effect, and will be removed in a future release.

Migration guide:

  • Drop any use_statistics_registry setting.

  • To use statistics providers, register them on the session with SessionStateBuilder::with_statistics_registry(...). To keep the built-in NDV-aware providers the flag previously enabled, register StatisticsRegistry::default_with_builtin_providers().

  • To opt out, register nothing (the default).

Generated protobuf JsonWriterOptions gained a compression_level field#

The generated public protobuf JsonWriterOptions struct now carries an optional compression_level so JSON sink plans preserve explicitly configured compression levels across serialization.

Who is affected:

  • Users constructing generated JsonWriterOptions values with an exhaustive struct literal.

Migration guide:

Set compression_level explicitly, or fill it from Default:

// Before
JsonWriterOptions { compression }

// After
JsonWriterOptions {
    compression,
    compression_level: None,
}
// or
JsonWriterOptions {
    compression,
    ..Default::default()
}

The protobuf wire format remains backward compatible.

See PR #24945 for details.

DefaultStatisticsProvider is deprecated#

datafusion_physical_plan::operator_statistics::DefaultStatisticsProvider is deprecated. It is redundant: the statistics walk (StatisticsContext) falls back to each operator’s statistics_from_inputs when the provider chain delegates or is empty, so a terminal “default” provider is no longer needed. It is no longer part of StatisticsRegistry::default_with_builtin_providers().

Migration guide:

  • Remove DefaultStatisticsProvider from any custom provider chain; register no terminal provider instead (the walk falls back on its own).

StatisticsRegistry::compute and compute_base are deprecated#

Use the walk instead:

StatisticsContext::new_with_registry(registry)
    .compute_extended(plan, &StatisticsArgs::new())?; // or .compute(...) for core Statistics

API change for floor and ceil UDF#

The output type of the floor and ceil UDFs has been changed from the exact input type to a rescaled type with the same bit width. For example, for input Decimal32(7,2) floor now returns Decimal32(6,0), where the new precision is p - s + 1, matching Spark’s behaviour. See [#24703] for more details.

Who is affected:

  • Users storing query result with these UDFs in a fixed schema

  • Users relying on arrow_typeof for these UDFs

Migration guide:

Change the expected type or wrap the expression in CAST. It’s recommended to avoid relying on decimal’s exact precision and scale.

map_extract / element_at return an empty list for absent keys#

map_extract (and its alias element_at) previously returned a single-element list containing NULL when the key was not present in the map. It now returns an empty list, matching the documented behavior and DuckDB. Two related cases changed at the same time, also matching DuckDB:

  • A NULL map input now yields NULL instead of [NULL].

  • A NULL lookup key now yields [] instead of [NULL].

A key that is present with a NULL value still returns [NULL], so absent keys and NULL values are now distinguishable.

Migration guide:

-- Before
SELECT map_extract(MAP {'a': 1}, 'missing');  -- [NULL]

-- After
SELECT map_extract(MAP {'a': 1}, 'missing');  -- []

Expressions that assumed the result always has exactly one element, for example by checking its length, unnesting it, or comparing it to [NULL], should treat an empty list as the absent-key case instead.

See issue #24981 and issue #24983 for details.

DataFrame::from_columns accepts IntoIterator#

DataFrame::from_columns now accepts any IntoIterator<Item = (&str, ArrayRef)> instead of specifically accepting a Vec<(&str, ArrayRef)>.

// Existing Vec usage continues to work
let df = DataFrame::from_columns(vec![
    ("id", id),
    ("name", name),
])?;

// Arrays can now be used directly
let df = DataFrame::from_columns([
    ("id", id),
    ("name", name),
])?;

Most existing call sites using Vec require no changes. Code that relies on the exact non-generic function signature of DataFrame::from_columns may need to be updated to account for the new generic API.

Physical filter pushdown resolves columns by position#

datafusion_physical_plan::filter_pushdown::ChildFilterDescription::from_child and FilterDescription::from_children now resolve filter columns by position instead of looking them up by name. This prevents incorrect results when a child schema contains duplicate column names. from_child requires the child field at each referenced position to have the same name as the filter column.

ChildFilterDescription::from_child_with_allowed_indices is deprecated but preserves its previous name-based mapping to the first matching child field. Migrate to from_child_with_column_mapping because name resolution is ambiguous when the child schema contains duplicate field names.

Migration guide:

Use from_child (or from_children for multiple children) when the parent and child schemas have matching column positions and names. When a node projects, reorders, or renames columns, use from_child_with_column_mapping with an explicit map from parent output indices to child input indices. Columns absent from the mapping cannot be pushed down.

For example, if the parent outputs [a, b] and the child outputs [b, a], a filter on a@0 must map to child column a@1:

use std::collections::{HashMap, HashSet};
use datafusion_physical_plan::filter_pushdown::ChildFilterDescription;

// Before: allow parent column 0 and resolve "a" by name in the child.
let description = ChildFilterDescription::from_child_with_allowed_indices(
    &parent_filters,
    HashSet::from([0]),
    &child,
)?;

// After: explicitly map parent column 0 to child column 1.
let description = ChildFilterDescription::from_child_with_column_mapping(
    &parent_filters,
    HashMap::from([(0, 1)]),
    &child,
)?;