Upgrade Guides#

Ballista 54.0.0#

Note: Ballista 54.0.0 has not been released yet. The information provided in this section pertains to features and changes that have already been merged to the main branch and are awaiting release in this version.

For library and embedding users#

Upgrade to DataFusion 54#

Ballista 54.0.0 upgrades its DataFusion dependency to 54.0.0. Code built on ballista-core / DataFusion APIs must move with it. See DataFusion’s own Upgrade Guide for 54.0.0 for the DataFusion-side API changes.

`job_id` is now a newtype#

Scheduler job identifiers are no longer bare Strings. ballista-core now exports a transparent JobId newtype (ballista_core::JobId) so a job id can no longer be confused with another string identifier at the type level. Public signatures that previously carried a String/&str job id now carry a JobId / &JobId — for example DistributedQueryExec::job_id() now returns Option<JobId> instead of Option<String>. Construct one with JobId::new(...) or .into(), and recover the inner string with .as_str() or .into_inner().

// before
let id: Option<String> = query_exec.job_id();

// after
use ballista_core::JobId;
let id: Option<JobId> = query_exec.job_id();
let id_str: Option<String> = id.map(|id| id.into_inner());

For cluster operators#

CORS origins and methods are configured via CLI flags#

The scheduler REST API no longer reads CORS configuration from environment variables. The BALLISTA_CORS_ALLOWED_ORIGINS and BALLISTA_CORS_ALLOWED_METHODS environment variables are removed and replaced by the --cors-allowed-origins and --cors-allowed-methods scheduler CLI flags (each a comma-separated list; * allows any). If you set these environment variables, move the values to the flags — otherwise the scheduler falls back to its defaults (origins http://localhost:8080 and https://nightlies.apache.org; methods GET, PATCH, OPTIONS).

# before (environment variables)
BALLISTA_CORS_ALLOWED_ORIGINS=https://example.com \
BALLISTA_CORS_ALLOWED_METHODS=GET,PATCH \
  ballista-scheduler

# after (scheduler CLI flags)
ballista-scheduler \
  --cors-allowed-origins https://example.com \
  --cors-allowed-methods GET,PATCH

Planner and execution behavior changes#

The static planner now broadcasts small join build sides#

Ballista previously disabled CollectLeft broadcast joins entirely (both datafusion.optimizer.hash_join_single_partition_threshold and hash_join_single_partition_threshold_rows defaulted to 0). In 54.0.0 those defaults are raised to 10 MB (10485760) and 1000000 rows, and the static DefaultDistributedPlanner now converts a SortMergeJoinExec whose smaller side fits under ballista.optimizer.broadcast_join_threshold_bytes (default 10 MB) into a broadcast CollectLeft hash join. This conversion is governed by the new ballista.optimizer.broadcast_sort_merge_join_enabled config key, which defaults to true.

The consequence is that join plans — and therefore performance and shuffle behavior — change on upgrade even with AQE off. Broadcast is applied only to join types that are safe to broadcast (inner and right-side variants); other join types remain repartitioned. To restore the previous behavior, set ballista.optimizer.broadcast_sort_merge_join_enabled = false, ballista.optimizer.broadcast_join_threshold_bytes = 0, datafusion.optimizer.hash_join_single_partition_threshold = 0, and hash_join_single_partition_threshold_rows = 0.

Serialized empty projections#

The plan serialization for a filter that projects to zero columns changed. In 53.x an empty projection (Some([])) and “all columns” (None) both encoded to an empty list and decoded back to None, shifting downstream column indices; the scheduler now rewrites such filters into a serde-safe form before serialization. Together with the DataFusion 54 protobuf bump, this means a 54.0.0 scheduler and a 53.x executor (or vice versa) cannot reliably exchange serialized plans. Upgrade schedulers and executors together — do not run a mixed-version cluster across this upgrade.

AQE join behavior (opt-in)#

If you have enabled ballista.planner.adaptive.enabled = true, adaptive join selection now delays the join decision and can promote a small build side to a broadcast (CollectLeft) hash join at runtime, and empty-join inputs are short-circuited by the propagate-empty rule. This changes the runtime join plans (and their results’ timing/shape) for adaptive clusters; it is gated by ballista.planner.adaptive_join.enabled (default true). Clusters using the default (AQE off) are unaffected.