Apache DataFusion Ballista 54.1.0 Released

Posted on: Sun 09 August 2026 by pmc

We are pleased to announce version 54.1.0 of Apache DataFusion Ballista. Ballista is a distributed query execution engine that enhances Apache DataFusion by enabling parallel execution of workloads across multiple nodes.

This is a patch release, consisting of 10 commits from 3 contributors. It follows 54.0.0 and contains no new features and no breaking changes — only an upgrade to DataFusion 54.1.0 and a set of bug fixes backported from main. Users running 54.0.0 are encouraged to upgrade.

Upgrade to DataFusion 54.1.0¶

Ballista 54.1.0 is built on DataFusion 54.1.0, picking up the fixes in that patch release. No API changes were required, so the 54.0.0 upgrade guide still applies for anyone moving from an earlier release.

Cluster stability fixes¶

Three fixes address problems that only appear in a real multi-node deployment:

• Executor client pool enabled by default. The executor's client_ttl defaulted to 0, which disabled the gRPC client pool entirely. Without a pool, the shuffle-read path opened and discarded a TCP connection per fetch, so a single shuffle-heavy query could exhaust the host's ephemeral ports and fail with AddrNotAvailable — as would every query after it, until those sockets left TIME_WAIT. A single-executor cluster never showed this, because shuffle reads are served from local files. client_ttl now defaults to 30 seconds, so the pool is on unless explicitly disabled.
• Executor startup race. The executor spawned its gRPC server and registered with the scheduler immediately afterwards, but the socket was not bound yet when the scheduler dialed back. Losing that race produced ECONNREFUSED, a failed registration, and an executor that died at startup. The listener is now bound before registration can run.
• Executor shutdown. The shutdown notification is now sent before the notifier is dropped, so a graceful shutdown is actually observed by the components waiting on it.

Task retry and shuffle memory¶

• Retryable IO errors on a join's build side. A retryable IO error raised on a join's shared build side arrives at the failed-task classifier wrapped in DataFusionError::Shared, so a match on the outermost variant never saw the underlying IoError and the task was marked non-retryable. Classification now operates on the root error, regardless of wrapping.
• Sort shuffle writer respects memory pool pressure. The sort shuffle writer discarded the result of try_grow on its memory reservation, so a rejected grow produced no spill: the batch stayed buffered while the shared MemoryPool had not accounted for it, and other consumers saw memory as free that was not. A rejected grow is now treated as a spill trigger alongside the existing per-task budget.

Planning correctness¶

• GROUP BY-less aggregates are preserved when the scheduler propagates empty stages, so a query whose input produces no rows still returns the correct single-row aggregate result rather than nothing.
• EmptyExec partition counts survive stage serialization, fixing plans whose partitioning changed as a stage was sent to an executor.
• Task partitions are mapped through UnionExec when restricting file scans, so each task scans the files it was assigned.
• SortMergeJoinExec broadcast conversion is now disabled by default. The static-planner optimization that broadcasts the small build side of a sort-merge join, added in 54.0.0, is opt-in in this release while outstanding issues are resolved.

Thank You¶

Thanks to Andy Grove, Alexander Domenti, and goingforstudying-ctrl for the commits in this release, and to everyone who contributed by filing issues, reviewing PRs, and providing feedback. Full details are in the changelog.