Upgrade Guides

DataFusion 57.0.0

Note: DataFusion 57.0.0 has not been released yet. The information provided in this section pertains to features and changes that have already been merged to the main branch and are awaiting release in this version.

Substrait consumers use explicit scope context

Add a final &SubstraitConsumerContext argument to custom relation and expression callbacks on SubstraitConsumer. Forward it when calling conversion helpers or consuming child relations and expressions.

Remove implementations of push_outer_schema, pop_outer_schema, get_outer_schema, push_lambda_parameters, pop_lambda_parameters, and lambda_variable, and remove embedded DefaultSubstraitLambdaConsumer values. Use the context’s get_outer_schema and lambda_variable methods to resolve outer references and lambda parameters. To enter a scope, derive a child with context.with_outer_schema(...) or context.with_lambda_parameters(...) and pass it to nested conversions. No pop operation is needed; the parent stays unchanged.

from_substrait_plan and from_substrait_plan_with_consumer create the root context. For a new conversion started directly through a helper, pass &SubstraitConsumerContext::default(). Nested conversions must receive the current context.

Custom scoped values can be stored with context.with_extension(Arc::new(value)) and retrieved with context.get_extension::<MyScope>(). Pass the returned child context to nested conversions. Children inherit values; replacing a value leaves parents and siblings unchanged. The values themselves are shared through Arc.

Update logical-plan snapshots if needed: generated lambda parameter names are local to lexical scopes, so sibling lambdas may reuse names.