Apache DataFusion Ballista 55.0.0 Changelog#

This release consists of 322 commits from 25 contributors. See credits at the end of this changelog for more information.

Fixed bugs:

  • fix(scheduler): persist terminal job state before cache eviction #2037 (phillipleblanc)

  • fix: spill sort shuffle writer when the memory pool rejects a reservation grow #2061 (andygrove)

  • fix: enable the executor client pool by default #2069 (andygrove)

  • fix: map a task’s partition through UnionExec when restricting file scans #2070 (andygrove)

  • fix: keep EmptyExec partition count intact across stage serialization [AQE] #2062 (andygrove)

  • fix: remove the static planner SortMergeJoinExec broadcast conversion #2072 (andygrove)

  • fix: propagate sort shuffle memory_limit_per_task_bytes to executors #2090 (andygrove)

  • fix: executor hanging on ctrl+c #2107 (sandugood)

  • fix: Various fixes for AQE join selection #2124 (andygrove)

  • fix: Enable more TPC-DS queries #2064 (andygrove)

  • fix(executor): keep heartbeating while all task slots are busy #2159 (andygrove)

  • fix: unskip five TPC-DS queries by rewriting generator column names #2183 (andygrove)

  • fix: stop every task running every union branch at low partition counts #2186 (andygrove)

  • fix(scheduler): execute null-aware anti joins in one task #2188 (phillipleblanc)

  • fix(scheduler): keep GROUP BY-less aggregates when propagating empty stages #2194 (andygrove)

  • fix(scheduler): harden PropagateEmptyExecRule against grouping sets, embedded projections, and dropped exchange partitioning #2200 (andygrove)

  • fix(cli): sort jobs by stage completion ratio #1947 (QuakeWang)

  • fix(scheduler): preserve stages on transition errors #2215 (QuakeWang)

  • fix(scheduler): fail running jobs when all executors are lost #2212 (akshaychitneni)

  • fix(executor): bind the gRPC listener before registering with the scheduler #2225 (andygrove)

  • fix: update mTLS example for poll_loop arguments #2229 (phillipleblanc)

  • fix(scheduler): route launch-failure executor removal through ExecutorLost #2240 (andygrove)

  • fix: classify wrapped FetchFailed as fetch partition error #2202 (villebro)

  • fix(core): fix main build errors #2254 (avantgardnerio)

  • fix: pin datafusion versions to 54.0.0 for python subproject #2277 (milenkovicm)

  • fix(scheduler): key AQE shuffle output by job_id #2278 (villebro)

  • fix(scheduler): default no-executors grace to executor timeout (>= heartbeat) #2279 (akshaychitneni)

  • fix(docs): k8s executors dial the registration Service; scheduler needs maxUnavailable: 1 #2289 (avantgardnerio)

  • fix(ci): apply cargo fmt to chaos-executor #2304 (andygrove)

  • fix(scheduler): label every operator in the DOT graph #2305 (andygrove)

  • fix(core): report the real partitioning from ShuffleWriterExec #2307 (andygrove)

  • fix(scheduler): drop an executor’s cached gRPC client when it is removed #2310 (Dandandan)

  • fix: repair AQE plan rewrites, remove a redundant shuffle, and enable adaptive query planning by default #2315 (Dandandan)

  • fix(tui): use the shared API wire types instead of local copies #2258 (andygrove)

  • fix(chaos): harden k8s backend — supervise port-forward, guard fixture dir #2346 (akshaychitneni)

  • fix(docker): base runtime images on ubuntu:26.04 #2380 (avantgardnerio)

  • fix(scheduler): fix test job-completion polling #2354 (NoahKusaba)

  • fix(ci): pin tinyvec in the MSRV dependency checker #2421 (andygrove)

  • fix(executor): wake graceful shutdown after tasks drain #2393 (1fanwang)

  • fix(scheduler): repair test broken by a stale-base merge #2439 (NoahKusaba)

  • fix: derive stage input/output rows from the plan’s leaves and root instead of summing every operator #2435 (Dandandan)

  • fix(ci): pull the MinIO test image from quay.io #2447 (comphead)

  • fix(ci): resolve datafusion.proto from datafusion-proto-models #2390 (AnuragRaut08)

  • fix(ci): upload both cluster logs from the container benchmark jobs #2460 (amitvijapur)

  • fix(ci): run the S3 integration tests against RustFS instead of MinIO #2488 (NoahKusaba)

  • fix(executor): forward arrow-ipc-optimizations to ballista-core #2373 (stag7824)

  • fix(scheduler): refund the vcores of running tasks when a job is aborted #2477 (amitvijapur)

  • fix(scheduler): fail jobs whose tasks cannot be prepared #2492 (andygrove)

  • fix(scheduler): declare distributed EXPLAIN output columns not-null #2494 (andygrove)

  • fix(core): refuse queries that mix information_schema with other tables #2493 (andygrove)

  • fix(release): prepare the 55 release process #2527 (andygrove)

  • fix(docker): publish images from release tag pushes #2526 (andygrove)

Performance related:

  • perf(kll): batch-oriented ingest KLL #2216 (avantgardnerio)

  • perf(core): zero-copy block receive on the remote fetch path #2312 (Dandandan)

  • perf(core): push a consumer’s row limit into RangeShuffleReaderExec #2334 (Dandandan)

  • perf(scheduler): push filters once, not on every replan #2344 (Dandandan)

  • perf(scheduler): only give a broadcast build side its own stage when the probe is partitioned #2342 (Dandandan)

  • perf(scheduler): read an unfiltered small scan inline instead of staging it #2338 (Dandandan)

  • perf(core): write one sort-shuffle file per task instead of one per input #2316 (Dandandan)

  • perf(core): reuse shuffle partition-index buffers across batches #2359 (NoahKusaba)

  • perf(core): strength reduce hash partition modulo #2363 (NoahKusaba)

  • perf(core): skip IPC validation on the sort-shuffle read paths #2365 (NoahKusaba)

  • perf(scheduler): stop re-encoding the session config for every task #2358 (NoahKusaba)

  • perf(core): remove the inert shuffle-read sort #2414 (NoahKusaba)

  • perf(scheduler): share the file statistics cache across sessions #2498 (andygrove)

Implemented enhancements:

  • feat: make compression codec configurable #2039 (sandugood)

  • feat: make broadcast_join_threshold_bytes/rows authoritative under AQE and the static planner #2086 (andygrove)

  • feat: add BALLISTA_PROTOCOL_VERSION + k8s health probes #2088 (avantgardnerio)

  • feat: allow an external semaphore for executor task concurrency #1892 (Jeadie)

  • feat: log sort shuffle spills at WARN, keep write completion at DEBUG #2091 (andygrove)

  • feat(benchmarks): persist TPC-H results resiliently and add a compare subcommand #2102 (andygrove)

  • feat(core): RuntimeStatsExec — passthrough tap for row counts + quantile sketches #2094 (avantgardnerio)

  • feat(core): BufferExec — generic flow-control op with a memory-pool Dam mode #2095 (avantgardnerio)

  • feat: Improve TPC-H performance with AQE on #2084 (andygrove)

  • feat(benchmarks): add –skip to omit queries from a tpch suite run #2129 (andygrove)

  • feat(core): report the actual sort shuffle spill trigger in logs #2143 (andygrove)

  • feat(core): add UnorderedRangeRepartitionExec — value-range router over unordered inputs #2123 (avantgardnerio)

  • feat(executor): default to a bounded, auto-sized memory pool #2160 (andygrove)

  • feat(core): add OrderedRangeRepartitionExec — N sorted overlapping → K sorted disjoint #2169 (avantgardnerio)

  • feat(core): add PerPartitionFilterExec — per-input-partition boolean predicate #2195 (avantgardnerio)

  • feat(core): rewrite NOT IN subqueries into distributable anti join plans #2199 (andygrove)

  • feat(scheduler): range-repartition rule primitives (batteries-included) #2196 (avantgardnerio)

  • feat: make gRPC max message size consistently configurable #2173 (andygrove)

  • feat: add poll_now_notify to poll_loop and on_work_available callback #1893 (Jeadie)

  • feat(scheduler): expose existing job state event stream #1891 (Jeadie)

  • feat(scheduler): Redirect ‘/’ to nighlies.a.o WebTUI #2059 (martin-g)

  • feat(core): metrics() for OrderedRangeRepartitionExec and RuntimeStatsExec #2250 (avantgardnerio)

  • feat: add DistributedQueryExec::explain_executed_plan and show logical plan in display #2042 (andygrove)

  • feat(core): PBWAG — parallel BWAG wrapper #2253 (avantgardnerio)

  • feat(core): RangeShuffleReaderExec — ordering-preserving shuffle reader (purely additive) #2255 (avantgardnerio)

  • feat(core): RangeFilterExec — sorted-input per-partition range filter (purely additive) #2262 (avantgardnerio)

  • feat(history): add the event-log schema, writer, and reader #2260 (andygrove)

  • feat(scheduler): record a per-job event log behind –event-log-dir #2264 (andygrove)

  • feat: pluggable job id generator #2269 (milenkovicm)

  • feat: parallel BWAG for the range-window shape (h2o Q8) #2223 (avantgardnerio)

  • feat: embedded flight proxy explicit config entry #2288 (sandugood)

  • feat(core): quantile sketching for any fixed-width ORDER BY key #2294 (avantgardnerio)

  • feat(core): Replace TDigest with KLL #2337 (avantgardnerio)

  • feat: parallel prefix scan for UNBOUNDED PRECEDING window aggregates #2211 (avantgardnerio)

  • feat: remove serialisation of built-in file formats from ballista #2348 (milenkovicm)

  • feat(core): range shuffle reads only the bytes a consumer’s range covers #2364 (avantgardnerio)

  • feat(scheduler): add the history server #2265 (andygrove)

  • feat: scheduler REST & GRPC returns ballista version headers #2378 (milenkovicm)

  • feat: cleanup & simplify ballista config #2387 (milenkovicm)

  • feat(history): mark running event logs with a .running suffix until job completion #2405 (milenkovicm)

  • feat(scheduler): stage a join’s build side before deciding the join #2434 (andygrove)

  • feat(scheduler): document REST API with OpenAPI and serve /api/openapi.json #2397 (DebadityaHait)

  • feat(scheduler): record scheduler id in job starts #2462 (villebro)

  • feat(scheduler): add a setting to disable the shared file statistics cache #2510 (andygrove)

  • feat: Column statistics support v1 #1916 (coderfender)

  • feat: error when the client and scheduler have different major versions #2514 (andygrove)

  • feat: Arrow Flight SQL frontend for the scheduler + ADBC support #2416 (andygrove)

Documentation updates:

  • docs: add user personas as a review contract for additive changes #2002 (andygrove)

  • docs: include TestPyPI wheels link in release vote email #2010 (andygrove)

  • docs: remove outdated TPC-H benchmark results from READMEs #2015 (andygrove)

  • Rename task_slots -> vcores (executor spec + CLI) #2045 (avantgardnerio)

  • docs: add benchmarking page to the contributors guide #2075 (andygrove)

  • Multi-partition tasks: partition_slice plumbing + SortShuffleWriter refactor #2038 (avantgardnerio)

  • docs(config): rewrite max_partitions_per_task description to be timeless #2093 (avantgardnerio)

  • docs: refresh SF1000 benchmark results on 2x16-core AQE-on reference cluster #2109 (andygrove)

  • Remove the hash-partitioning shuffle writer #2106 (andygrove)

  • docs: correct scan partitioning description in tuning guide #2125 (andygrove)

  • Generate the config reference documentation from the config registry #2146 (andygrove)

  • docs: refresh benchmarking.md with current SF1000 results (Ballista main @ 696ca29b, Spark 3.4 baseline) #2161 (andygrove)

  • docs: refresh benchmarking.md with dedicated shuffle PVC results #2171 (andygrove)

  • docs: add 54.0.0 changelog #2012 (andygrove)

  • feat(core,scheduler): transport RuntimeStatsExec reports to scheduler; log merged quantile cuts per stage #2175 (avantgardnerio)

  • docs(kubernetes): two-Service pattern for scheduler bootstrap #2245 (avantgardnerio)

  • docs: add 54.1.0 changelog to main #2271 (andygrove)

  • docs: document Web TUI #2268 (jvsilva12600009)

  • docs: clean up stale and broken documentation links #2287 (AntiO2)

  • docs: document DataFusion dependency cycle in contributor guide #2293 (andygrove)

  • docs: explain why Ballista uses a blocking shuffle #2308 (andygrove)

  • docs(bench): refresh TPC-H SF1000 results at 67b3a19b #2417 (andygrove)

  • docs: move developer docs into the published contributors guide #2481 (NoahKusaba)

  • docs: say vcores instead of task slots and concurrency levels #2482 (NoahKusaba)

  • docs: include examples and config tables from source instead of copying them #2480 (NoahKusaba)

  • docs: describe how joins are chosen under AQE #2479 (NoahKusaba)

  • ci: run ruff over the whole repository #2472 (NoahKusaba)

  • docs(chaos): don’t link to the feature-gated k8s module #2506 (milenkovicm)

  • docs: correct stale defaults, flags, and feature docs #2478 (NoahKusaba)

  • chore(release): release the Python client separately from the Rust crates #2512 (andygrove)

Other:

  • ci: install Rust via rustup in wheel build to fix release-build startup_failure #2008 (andygrove)

  • chore: Update project description to remove Apache Arrow #2044 (andygrove)

  • chore(deps): bump cryptography from 46.0.3 to 48.0.1 in /python #2056 (dependabot[bot])

  • chore(deps): bump pyjwt from 2.12.0 to 2.13.0 in /python #2055 (dependabot[bot])

  • chore(ci): Add a CI workflow that checks that the used CI actions and their versions are allowed #2058 (martin-g)

  • refactor(tui): Drop the header banner/logo #2057 (martin-g)

  • Add TPC-DS correctness testing #2048 (andygrove)

  • ci: skip expensive workflows for docs-only changes #2073 (andygrove)

  • chore(scheduler,executor): downgrade #2038 per-task traces to debug #2104 (avantgardnerio)

  • ci(tpch): add AQE-on smoke pass with multi-partition tasks #2120 (avantgardnerio)

  • Retryable IO errors misclassified when wrapped in DataFusionError::Shared #2119 (goingforstudying-ctrl)

  • chore: upgrade DataFusion to 54.1.0 #2126 (andygrove)

  • ci: allow the docs site deploy to be triggered manually #2177 (andygrove)

  • kll: add native Rust KLL quantile sketch #2180 (avantgardnerio)

  • chore(deps): bump arrow from 58.3.0 to 58.4.0 #2178 (andygrove)

  • chore(deps): bump quinn-proto from 0.11.14 to 0.11.16 #2176 (dependabot[bot])

  • ci: run TPC-DS SF1 with 1 and 4 partitions per task #2197 (andygrove)

  • chore: use taplo for Cargo.toml formatting #2179 (villebro)

  • test: add HA chaos harness that drives the scheduler’s fault-tolerance paths on a real cluster #2026 (andygrove)

  • chore(deps): bump cryptography from 48.0.1 to 50.0.0 in /python #2221 (dependabot[bot])

  • ci: add h2o window suite CI job #2230 (avantgardnerio)

  • chore(docker): ca-certificates + h2o binary in benchmarks image #2251 (avantgardnerio)

  • refactor(scheduler): extract REST API wire types into ballista-api-types #2256 (andygrove)

  • refactor(core): use DataFusionError::External for structured task errors #2276 (villebro)

  • chore(deps): Track DataFusion git, pinned to the 55.0.0-rc2 tag #2241 (andygrove)

  • chore(deps): bump the DataFusion pin to the 55.0.0-rc3 tag #2301 (andygrove)

  • test: centralize physical plan snapshot assertions #2297 (floze-the-genius)

  • test(chaos): add a Kubernetes (kind) backend for the chaos harness #2244 (akshaychitneni)

  • chore: prune stale NOTICE.txt entries, add mini-redis attribution #2300 (andygrove)

  • chore(core): remove unused diagram module #2355 (NoahKusaba)

  • Revert “remove redundant logic from ballista logical codec (#2348)” #2379 (andygrove)

  • chore(deps): use the released DataFusion 55.0.0 crates from crates.io #2375 (andygrove)

  • ci: assign issues via take/untake comments #2391 (andygrove)

  • chore: require an approving review before merging #2394 (andygrove)

  • refactor(scheduler): share the constructor tail between SchedulerServer builders #2398 (NoahKusaba)

  • chore(deps): drop unused md-5, move insta to dev-deps #2407 (NoahKusaba)

  • chore: consolidate shared manifest metadata into the workspace #2409 (NoahKusaba)

  • test: dedupe shuffle reader test setup #2415 (NoahKusaba)

  • chore(ci): run cargo machete to catch unused dependencies #2411 (7487)

  • ci: share one cluster harness across the tpch, tpcds and h2o jobs #2408 (NoahKusaba)

  • bug: Prevent exec hang when task decode fails #2016 (coderfender)

  • chore(deps): bump DataFusion to 55.1.0 #2461 (andygrove)

  • ci: move the stale PR workflow where GitHub runs it #2475 (NoahKusaba)

  • ci: fix lint scripts that skipped checks or passed without running #2473 (NoahKusaba)

  • chore: remove leftovers inherited from Arrow #2474 (NoahKusaba)

  • refactor: move the Flight proxy service into ballista-core #2495 (andygrove)

  • test(chaos): run #2029 executor-loss scenarios on the k8s backend #2425 (akshaychitneni)

  • chore(deps): bump pyjwt from 2.13.0 to 2.15.0 in /python #2515 (dependabot[bot])

  • chore(ci): fix clippy errors from Rust 1.99 #2519 (NoahKusaba)

  • chore: bump Ballista version to 55.0.0 #2524 (andygrove)

  • chore(deps): bump urllib3 from 2.7.0 to 2.8.0 in /python #2516 (dependabot[bot])

Credits#

Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.

   135	dependabot[bot]
    79	Andy Grove
    30	Brent Gardner
    24	NoahKusaba
     9	Daniël Heres
     7	Marko Milenković
     5	Akshay Chitneni
     5	Ville Brofeldt
     3	Jack Eadie
     3	Martin Grigorov
     3	Phillip LeBlanc
     3	alexander domenti
     2	Amit Vijapur
     2	Bhargava Vadlamani
     2	QuakeWang
     1	Anurag Tryambak Raut
     1	BingEdward
     1	Floze
     1	Joao Victor e Silva
     1	Li Zinuo
     1	Oleks V
     1	Stefan Wang
     1	Syed Talha Ahmed Gardazi
     1	debaditya
     1	goingforstudying-ctrl

Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.