# Upgrade Guides ## Ballista 54.0.0 **Note:** Ballista `54.0.0` has not been released yet. The information provided in this section pertains to features and changes that have already been merged to the main branch and are awaiting release in this version. ### For library and embedding users #### Upgrade to DataFusion 54 Ballista `54.0.0` upgrades its DataFusion dependency to `54.0.0`. Code built on `ballista-core` / DataFusion APIs must move with it. See DataFusion's own [Upgrade Guide for 54.0.0](https://datafusion.apache.org/library-user-guide/upgrading/54.0.0.html) for the DataFusion-side API changes. #### `job_id` is now a newtype Scheduler job identifiers are no longer bare `String`s. `ballista-core` now exports a transparent `JobId` newtype (`ballista_core::JobId`) so a job id can no longer be confused with another string identifier at the type level. Public signatures that previously carried a `String`/`&str` job id now carry a `JobId` / `&JobId` — for example `DistributedQueryExec::job_id()` now returns `Option` instead of `Option`. Construct one with `JobId::new(...)` or `.into()`, and recover the inner string with `.as_str()` or `.into_inner()`. ```rust // before let id: Option = query_exec.job_id(); // after use ballista_core::JobId; let id: Option = query_exec.job_id(); let id_str: Option = id.map(|id| id.into_inner()); ``` ### For cluster operators #### CORS origins and methods are configured via CLI flags The scheduler REST API no longer reads CORS configuration from environment variables. The `BALLISTA_CORS_ALLOWED_ORIGINS` and `BALLISTA_CORS_ALLOWED_METHODS` environment variables are removed and replaced by the `--cors-allowed-origins` and `--cors-allowed-methods` scheduler CLI flags (each a comma-separated list; `*` allows any). If you set these environment variables, move the values to the flags — otherwise the scheduler falls back to its defaults (origins `http://localhost:8080` and `https://nightlies.apache.org`; methods `GET`, `PATCH`, `OPTIONS`). ``` # before (environment variables) BALLISTA_CORS_ALLOWED_ORIGINS=https://example.com \ BALLISTA_CORS_ALLOWED_METHODS=GET,PATCH \ ballista-scheduler # after (scheduler CLI flags) ballista-scheduler \ --cors-allowed-origins https://example.com \ --cors-allowed-methods GET,PATCH ``` ### Planner and execution behavior changes #### The static planner now broadcasts small join build sides Ballista previously disabled `CollectLeft` broadcast joins entirely (both `datafusion.optimizer.hash_join_single_partition_threshold` and `hash_join_single_partition_threshold_rows` defaulted to `0`). In `54.0.0` those defaults are raised to `10 MB` (`10485760`) and `1000000` rows, and the static `DefaultDistributedPlanner` now converts a `SortMergeJoinExec` whose smaller side fits under `ballista.optimizer.broadcast_join_threshold_bytes` (default `10 MB`) into a broadcast `CollectLeft` hash join. This conversion is governed by the new `ballista.optimizer.broadcast_sort_merge_join_enabled` config key, which defaults to `true`. The consequence is that join plans — and therefore performance and shuffle behavior — change on upgrade **even with AQE off**. Broadcast is applied only to join types that are safe to broadcast (inner and right-side variants); other join types remain repartitioned. To restore the previous behavior, set `ballista.optimizer.broadcast_sort_merge_join_enabled = false`, `ballista.optimizer.broadcast_join_threshold_bytes = 0`, `datafusion.optimizer.hash_join_single_partition_threshold = 0`, and `hash_join_single_partition_threshold_rows = 0`. #### Serialized empty projections The plan serialization for a filter that projects to zero columns changed. In `53.x` an empty projection (`Some([])`) and "all columns" (`None`) both encoded to an empty list and decoded back to `None`, shifting downstream column indices; the scheduler now rewrites such filters into a serde-safe form before serialization. Together with the DataFusion 54 protobuf bump, this means a `54.0.0` scheduler and a `53.x` executor (or vice versa) cannot reliably exchange serialized plans. Upgrade schedulers and executors together — do not run a mixed-version cluster across this upgrade. #### AQE join behavior (opt-in) If you have enabled `ballista.planner.adaptive.enabled = true`, adaptive join selection now delays the join decision and can promote a small build side to a broadcast (`CollectLeft`) hash join at runtime, and empty-join inputs are short-circuited by the propagate-empty rule. This changes the runtime join plans (and their results' timing/shape) for adaptive clusters; it is gated by `ballista.planner.adaptive_join.enabled` (default `true`). Clusters using the default (AQE off) are unaffected.