Upgrade Guides

DataFusion 55.0.0

Note: DataFusion 55.0.0 has not been released yet. The information provided in this section pertains to features and changes that have already been merged to the main branch and are awaiting release in this version.

is_dynamic_physical_expr is deprecated

datafusion_physical_expr_common::physical_expr::is_dynamic_physical_expr is deprecated. It was a thin wrapper over snapshot_generation(expr) != 0 used to ask “does this predicate contain a dynamic filter?”.

Prefer asking the question directly against the concrete type. For a one-off check, downcast to DynamicFilterPhysicalExpr:

use datafusion_physical_expr::expressions::DynamicFilterPhysicalExpr;
use datafusion_common::tree_node::{TreeNode, TreeNodeRecursion};

let mut is_dynamic = false;
predicate.apply(|e| {
    if e.downcast_ref::<DynamicFilterPhysicalExpr>().is_some() {
        is_dynamic = true;
        Ok(TreeNodeRecursion::Stop)
    } else {
        Ok(TreeNodeRecursion::Continue)
    }
})?;

If you also need to know whether the dynamic filters can still change (and to be notified when they do), use the new DynamicFilterTracking / DynamicFilterTracker API in datafusion_physical_expr:

use datafusion_physical_expr::DynamicFilterTracking;

let tracking = DynamicFilterTracking::classify(&predicate);
if tracking.contains_dynamic_filter() {
    // worth re-evaluating the predicate at runtime
}

FilePruner::try_new no longer builds a pruner for static predicates without statistics

datafusion_pruning::FilePruner::try_new now returns None when the predicate is purely static and the file carries no usable column statistics, because such a pruner can never prune anything beyond what planning already did. Previously it returned Some whenever a statistics struct was present (the “is this worth pruning?” decision lived in the Parquet opener). Files with column statistics, and predicates that carry a dynamic filter, are unaffected.