Upgrade Guides#

DataFusion 56.0.0#

Note: DataFusion 56.0.0 has not been released yet. The information provided in this section pertains to features and changes that have already been merged to the main branch and are awaiting release in this version.

ForeignSession::create_physical_plan is unsupported#

ForeignSession::create_physical_plan no longer forwards to the library that owns the session. It now returns a NotImplemented error because forwarding can re-enter an installed foreign planner, and the execution-plan handle returned by the old callback cannot restore local Rust type identities for downcasting. The original FFI_SessionRef callback slot remains in place for ABI compatibility with DataFusion 55 consumers, but calling that callback returns the same error.

The session-owning library should instead export its original planner as a datafusion_ffi::query_planner::FFI_QueryPlanner before installing a foreign planner. The foreign planner can retain and invoke that handle to receive a serialized physical plan reconstructed with local type identities. See the datafusion_ffi::query_planner module documentation for the complete delegation pattern. ForeignSession::query_planner, optimize, and physical_optimizers continue to forward to the owning session across the FFI boundary.

GroupColumn now requires values_preserving#

Custom implementations of the public GroupColumn trait must implement values_preserving. This method returns selected rows without changing the stored values or their group indices. It preserves the requested order and supports repeated indices.

Migration guide:

Implement values_preserving, call selection.validate_num_groups(self.len())? before reading, and return rows in selection.iter() order without changing the builder state.

datafusion-proto: common option and constraint conversions are fallible#

Protobuf conversions for CsvOptions, JsonOptions, ParquetCdcOptions, Constraint, and Constraints now reject integer values that do not fit usize. Their infallible From implementations have been replaced with TryFrom. The existing TryFrom conversions for ParquetOptions and TableParquetOptions now also validate all usize-backed fields. A constraint without a constraint_mode returns an error instead of panicking.

Migration guide:

// Before
let csv = CsvOptions::from(&proto_csv);
let cdc = ParquetCdcOptions::from(proto_cdc);
let constraints: Constraints = proto_constraints.into();

// After
let csv = CsvOptions::try_from(&proto_csv)?;
let cdc = ParquetCdcOptions::try_from(proto_cdc)?;
let constraints = Constraints::try_from(proto_constraints)?;

See issue #24170 for details.

ExecutionOptions has a new enable_nlj_coordinated_fallback field#

ExecutionOptions gained a public enable_nlj_coordinated_fallback: bool field (default true). It controls whether the memory-limited NestedLoopJoinExec fallback shares per-chunk build-side state across probe partitions, which is what lets LEFT, LEFT SEMI, LEFT ANTI, LEFT MARK and FULL joins spill instead of failing with ResourcesExhausted when the right side has several partitions.

Who is affected:

  • Users constructing ExecutionOptions (or ConfigOptions) with an exhaustive struct literal. Reading or SET-ing configuration is unaffected.

  • Distributed engines that run each output partition as an independent task. The coordination assumes all probe partitions run in one process; with one coordinator per task the shared probe-thread counter never reaches zero and the fallback would stall. Such engines should set the flag to false, which keeps the previous fail-fast behaviour for the affected join types.

Migration guide:

Set the new field, or fill it from Default:

// Before
ExecutionOptions {
    batch_size: 8192,
    // ... every other field ...
}

// After: set it explicitly
ExecutionOptions {
    batch_size: 8192,
    enable_nlj_coordinated_fallback: true,
    // ... every other field ...
}

// After: or let the remaining fields come from Default, which also keeps
// future field additions from breaking the literal
ExecutionOptions {
    batch_size: 8192,
    ..Default::default()
}

Distributed engines opting out:

let mut config = SessionConfig::new();
config.options_mut().execution.enable_nlj_coordinated_fallback = false;

datafusion.optimizer.use_statistics_registry is deprecated and ignored#

The datafusion.optimizer.use_statistics_registry config flag is deprecated and now ignored (setting it emits a deprecation warning). The pluggable StatisticsRegistry is always consulted during the single statistics walk: providers registered on the session take effect directly. With no providers registered the registry is a no-op, so default behavior is unchanged from the previous default (use_statistics_registry = false).

Who is affected:

  • Anyone who set datafusion.optimizer.use_statistics_registry. The setting is still accepted (with a deprecation warning) but has no effect, and will be removed in a future release.

Migration guide:

  • Drop any use_statistics_registry setting.

  • To use statistics providers, register them on the session with SessionStateBuilder::with_statistics_registry(...). To keep the built-in NDV-aware providers the flag previously enabled, register StatisticsRegistry::default_with_builtin_providers().

  • To opt out, register nothing (the default).

Generated protobuf JsonWriterOptions gained a compression_level field#

The generated public protobuf JsonWriterOptions struct now carries an optional compression_level so JSON sink plans preserve explicitly configured compression levels across serialization.

Who is affected:

  • Users constructing generated JsonWriterOptions values with an exhaustive struct literal.

Migration guide:

Set compression_level explicitly, or fill it from Default:

// Before
JsonWriterOptions { compression }

// After
JsonWriterOptions {
    compression,
    compression_level: None,
}
// or
JsonWriterOptions {
    compression,
    ..Default::default()
}

The protobuf wire format remains backward compatible.

See PR #24945 for details.

DefaultStatisticsProvider is deprecated#

datafusion_physical_plan::operator_statistics::DefaultStatisticsProvider is deprecated. It is redundant: the statistics walk (StatisticsContext) falls back to each operator’s statistics_from_inputs when the provider chain delegates or is empty, so a terminal “default” provider is no longer needed. It is no longer part of StatisticsRegistry::default_with_builtin_providers().

Migration guide:

  • Remove DefaultStatisticsProvider from any custom provider chain; register no terminal provider instead (the walk falls back on its own).

StatisticsRegistry::compute and compute_base are deprecated#

Use the walk instead:

StatisticsContext::new_with_registry(registry)
    .compute_extended(plan, &StatisticsArgs::new())?; // or .compute(...) for core Statistics

API change for floor and ceil UDF#

The output type of the floor and ceil UDFs has been changed from the exact input type to a rescaled type with the same bit width. For example, for input Decimal32(7,2) floor now returns Decimal32(6,0), where the new precision is p - s + 1, matching Spark’s behaviour. See [#24703] for more details.

Who is affected:

  • Users storing query result with these UDFs in a fixed schema

  • Users relying on arrow_typeof for these UDFs

Migration guide:

Change the expected type or wrap the expression in CAST. It’s recommended to avoid relying on decimal’s exact precision and scale.

map_extract / element_at return an empty list for absent keys#

map_extract (and its alias element_at) previously returned a single-element list containing NULL when the key was not present in the map. It now returns an empty list, matching the documented behavior and DuckDB. Two related cases changed at the same time, also matching DuckDB:

  • A NULL map input now yields NULL instead of [NULL].

  • A NULL lookup key now yields [] instead of [NULL].

A key that is present with a NULL value still returns [NULL], so absent keys and NULL values are now distinguishable.

Migration guide:

-- Before
SELECT map_extract(MAP {'a': 1}, 'missing');  -- [NULL]

-- After
SELECT map_extract(MAP {'a': 1}, 'missing');  -- []

Expressions that assumed the result always has exactly one element, for example by checking its length, unnesting it, or comparing it to [NULL], should treat an empty list as the absent-key case instead.

See issue #24981 and issue #24983 for details.

DataFrame::from_columns accepts IntoIterator#

DataFrame::from_columns now accepts any IntoIterator<Item = (&str, ArrayRef)> instead of specifically accepting a Vec<(&str, ArrayRef)>.

// Existing Vec usage continues to work
let df = DataFrame::from_columns(vec![
    ("id", id),
    ("name", name),
])?;

// Arrays can now be used directly
let df = DataFrame::from_columns([
    ("id", id),
    ("name", name),
])?;

Most existing call sites using Vec require no changes. Code that relies on the exact non-generic function signature of DataFrame::from_columns may need to be updated to account for the new generic API.