<!---
  Licensed to the Apache Software Foundation (ASF) under one
  or more contributor license agreements.  See the NOTICE file
  distributed with this work for additional information
  regarding copyright ownership.  The ASF licenses this file
  to you under the Apache License, Version 2.0 (the
  "License"); you may not use this file except in compliance
  with the License.  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing,
  software distributed under the License is distributed on an
  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  KIND, either express or implied.  See the License for the
  specific language governing permissions and limitations
  under the License.
-->

# Upgrade Guides

## Ballista 55.0.0

**Note:** Ballista `55.0.0` has not been released yet. The information provided
in this section pertains to features and changes that have already been merged
to the main branch and are awaiting release in this version.

### Planner and execution behavior changes

#### `SortMergeJoinExec` is no longer converted to a broadcast join

`54.0.0` added a static-planner rewrite that converted a `SortMergeJoinExec`
with a small build side into a broadcast `CollectLeft` hash join, governed by
`ballista.optimizer.broadcast_sort_merge_join_enabled` (default `true`). That
rewrite produced incorrect results for some plans, so both the rewrite and its
config key are removed in `55.0.0`.

A `SortMergeJoinExec` now always executes as a sort-merge join over
repartitioned inputs. Broadcast promotion still applies to a `HashJoinExec`
whose smaller side fits under `ballista.optimizer.broadcast_join_threshold_bytes`
(default `10 MB`), which is unchanged.

**This affects the default configuration.** Ballista sets
`datafusion.optimizer.prefer_hash_join = false` by default (see
[#1648](https://github.com/apache/datafusion-ballista/issues/1648)), so joins are
planned as `SortMergeJoinExec` unless you opt out. Under `54.0.0` those joins were
then silently converted to broadcast hash joins whenever one side fit under the
threshold, so expect join plan shape — and therefore performance and shuffle
behavior — to change on upgrade for any query with a join whose smaller side is
under `10 MB`.

Alongside the incorrect results, the conversion also worked against the reason
sort-merge join is the default. DataFusion's hash join has no spill support, so a
`CollectLeft` build side must fit in memory in every parallel task on an executor;
sort-merge join spills. Converting a sort-merge join into a broadcast hash join
reintroduced exactly the memory behavior that the `prefer_hash_join = false`
default exists to avoid.

**Action required:** the key `ballista.optimizer.broadcast_sort_merge_join_enabled`
no longer exists, and Ballista rejects unknown configuration keys. If you set it
(including to `false`, which was the workaround for the incorrect results), remove
it — otherwise the session fails with:

```text
configuration key `optimizer.broadcast_sort_merge_join_enabled` does not exist
```

If you relied on it being `false`, no replacement is needed: that is now the only
behavior. If you relied on it being `true` and want small joins to broadcast, set
`datafusion.optimizer.prefer_hash_join = true` so joins are planned as a
`HashJoinExec`, which remains eligible for broadcast promotion. Note this opts you
into the non-spilling hash join for _all_ joins in the session, not only the small
ones.