Upgrade Guides

Ballista 55.0.0

Note: Ballista 55.0.0 has not been released yet. The information provided in this section pertains to features and changes that have already been merged to the main branch and are awaiting release in this version.

Planner and execution behavior changes

SortMergeJoinExec is no longer converted to a broadcast join

54.0.0 added a static-planner rewrite that converted a SortMergeJoinExec with a small build side into a broadcast CollectLeft hash join, governed by ballista.optimizer.broadcast_sort_merge_join_enabled (default true). That rewrite produced incorrect results for some plans, so both the rewrite and its config key are removed in 55.0.0.

A SortMergeJoinExec now always executes as a sort-merge join over repartitioned inputs. Broadcast promotion still applies to a HashJoinExec whose smaller side fits under ballista.optimizer.broadcast_join_threshold_bytes (default 10 MB), which is unchanged.

This affects the default configuration. Ballista sets datafusion.optimizer.prefer_hash_join = false by default (see #1648), so joins are planned as SortMergeJoinExec unless you opt out. Under 54.0.0 those joins were then silently converted to broadcast hash joins whenever one side fit under the threshold, so expect join plan shape — and therefore performance and shuffle behavior — to change on upgrade for any query with a join whose smaller side is under 10 MB.

Alongside the incorrect results, the conversion also worked against the reason sort-merge join is the default. DataFusion’s hash join has no spill support, so a CollectLeft build side must fit in memory in every parallel task on an executor; sort-merge join spills. Converting a sort-merge join into a broadcast hash join reintroduced exactly the memory behavior that the prefer_hash_join = false default exists to avoid.

Action required: the key ballista.optimizer.broadcast_sort_merge_join_enabled no longer exists, and Ballista rejects unknown configuration keys. If you set it (including to false, which was the workaround for the incorrect results), remove it — otherwise the session fails with:

configuration key `optimizer.broadcast_sort_merge_join_enabled` does not exist

If you relied on it being false, no replacement is needed: that is now the only behavior. If you relied on it being true and want small joins to broadcast, set datafusion.optimizer.prefer_hash_join = true so joins are planned as a HashJoinExec, which remains eligible for broadcast promotion. Note this opts you into the non-spilling hash join for all joins in the session, not only the small ones.