Apache DataFusion Ballista 54.0.0 Changelog#

This release consists of 171 commits from 11 contributors. See credits at the end of this changelog for more information.

Fixed bugs:

  • fix(python): clear latent lint debt and gate wheel jobs on lint/tests #1749 (andygrove)

  • fix(python): exclude mimalloc from Python wheel to avoid macOS segfault #1769 (andygrove)

  • fix: handle task execution errors and panics in executor #1793 (milenkovicm)

  • fix: downgrade time version to fix build issue #1862 (milenkovicm)

  • fix: use DataFrame.name instead of hardcoded string in metaclass method wrapping #1866 (djanand)

  • fix: honor client datafusion.* session config overrides in scheduler planning #1902 (andygrove)

  • fix: Adaptive query planning fails with remote object stores: “No suitable object store found” #1962 (milenkovicm)

  • fix: satisfy clippy 1.97 useless_borrows_in_formatting #1978 (andygrove)

Performance related:

  • perf: avoid redundant TaskStatus clones in scheduler update_task_status #1983 (andygrove)

  • perf: prune irrelevant partitions task #1911 (coderfender)

Implemented enhancements:

  • feat: task_manager delegats physical plan creation to execution graph #1726 (milenkovicm)

  • feat: ShuffleReader shows upstream_stage #1767 (milenkovicm)

  • feat: runtime stage’s metrics rendering #1778 (sandugood)

  • feat: add /api/job/{job_id}/config endpoint to retrieve session configuration #1792 (killzoner)

  • feat(aqe): delay join decision & introduce broadcast join in AQE #1752 (milenkovicm)

  • feat: Add web rendering of the TUI application #1742 (martin-g)

  • feat(aqe): chaos monkey testing for ballista execution #1802 (milenkovicm)

  • feat(aqe): empty join handling #1723 (sandugood)

  • feat(tui): Show shimmer animation for “Queued” and “Running” jobs #1808 (martin-g)

  • feat(cli): use local time in tracing subscriber logging for ballista-cli #1824 (jiengup)

  • feat(scheduler): Use command line options for CORS origins/methods instead of env vars #1818 (martin-g)

  • feat(tui): Add configuration popup for jobs #1825 (jiengup)

  • feat(TUI): enable various plan rendering formats #1828 (sandugood)

  • feat(tui): Add theming support #1850 (martin-g)

  • feat: NewType pattern for job_id #1852 (JarroVGIT)

  • feat(tui): Show job failed status below the Jobs table #1871 (martin-g)

  • feat: broadcast small build sides via non-zero CollectLeft thresholds #1900 (andygrove)

  • feat: broadcast small build side of SortMergeJoinExec in the static planner #1904 (andygrove)

  • feat: Allow submitting a pre-built physical plan to the scheduler #1924 (andygrove)

  • feat: cancel running stages and tasks on job failure or cancellation #1903 (killzoner)

  • feat: support running TPC-H benchmarks against S3-compatible object storage #1942 (andygrove)

  • feat: bound shuffle fetch with a reduce-side in-flight governor #1951 (andygrove)

  • feat: improve session state creation for AQE planner #1963 (milenkovicm)

  • feat: delete intermediate shuffle files on job success #1982 (andygrove)

  • feat: Share read-side runtime state across a session’s tasks on the executor #1995 (andygrove)

Documentation updates:

  • docs(release): document PyPI publish process for python client #1610 (andygrove)

  • docs(release): cover TestPyPI verification and fix stale wheel-download notes #1760 (andygrove)

  • refactor: make use of async_trait::async_trait consistent across project #1848 (JarroVGIT)

  • chore(docs): Update docs website to bring inline with Datafusion style #1857 (JarroVGIT)

  • Improve changelog generator and split CHANGELOG.md into per-version files #1768 (andygrove)

  • docs: add versioned upgrade guide (54.0.0) #1939 (andygrove)

  • docs: document TPC-H plan-stability golden files and regeneration #1979 (andygrove)

  • docs: update companies using ballista #1992 (coderfender)

  • docs: link AQE limitations to tracking issues and add AQE contributor guide section #1990 (andygrove)

Other:

  • chore(deps): bump idna from 3.11 to 3.15 in /python #1737 (dependabot[bot])

  • chore(ci): Execute the CI workflows only when this is a push to main or a PR against any branch #1738 (martin-g)

  • minor: move aqe executions plans to a directory #1744 (milenkovicm)

  • ci: pin all third-party actions to ASF-allowlisted SHAs #1748 (andygrove)

  • ci: pin Windows wheel builder to windows-2022 #1751 (andygrove)

  • chore: bump python crate to 53.0.0 and use path deps for ballista crates #1750 (andygrove)

  • ci: produce real aarch64 wheel and upload sdist artifact #1753 (andygrove)

  • ci: only run Python Release Build on RC tags and matching PRs #1754 (andygrove)

  • ci: gate TPC-H workflow on rust-only path allow list #1755 (andygrove)

  • ci: gate Docker workflow on rust-only path allow list #1756 (andygrove)

  • ci: gate Rust workflow on workspace path allow list and rename to Ballista Rust #1757 (andygrove)

  • ci: use maturin sdist subcommand to avoid building a wheel #1759 (andygrove)

  • ci: gate CodeQL workflow on .github path changes #1758 (andygrove)

  • minor: Ballista showplan helper script #1761 (milenkovicm)

  • chore(deps): bump requests from 2.32.5 to 2.33.0 in /python #1781 (dependabot[bot])

  • chore(ci): Execute the “Dependencies” workflow on push to main and for PRs #1782 (martin-g)

  • chore(deps): bump astral-tokio-tar from 0.6.1 to 0.6.2 #1794 (dependabot[bot])

  • chore(deps): bump pygments from 2.19.2 to 2.20.0 in /python #1800 (dependabot[bot])

  • minor: code cleanup and log level changes #1805 (milenkovicm)

  • minor(aqe): backport few datafusion physical rules identified as not idempotent #1806 (milenkovicm)

  • minor(aqe): add plan_id to dynamic join #1809 (milenkovicm)

  • chore(docker): Use yellow_tripdata_2022-01.parquet from nightlies.apache.org #1817 (martin-g)

  • chore(ci): Use MacOS Github hosted runners for the small CI checks #1810 (martin-g)

  • chore(ci): Fix the Rust installer for MacOS in dependencies.yml #1833 (martin-g)

  • chore(ci): Drop Windows CI runs #1801 (martin-g)

  • chore(ci): Fix the name of a CI job dependency #1834 (martin-g)

  • chore(github): Enable “Update branch” button for PRs #1839 (martin-g)

  • chore(ci): Download macos aarch64 build of protoc #1835 (martin-g)

  • chore(tui): Upgrade Ratatui to 0.30.1 / Ratzilla to 0.3.1 #1840 (martin-g)

  • chore(ci): Run cargo check and cargo doc CI jobs on MacOS #1846 (martin-g)

  • chore: re enable approx distinct test #1844 (coderfender)

  • bug: fix empty projection serde ambiguity bug #1863 (coderfender)

  • chore(tui): Use lighter foreground colors for the dark theme. For better contrast #1872 (martin-g)

  • minor: remove redundant scheduler info loggers #1879 (milenkovicm)

  • chore(ci): Deploy WebTUI to nightlies.a.o #1888 (martin-g)

  • ci: run TPC-H SF10 under AQE off + on, and build it faster #1913 (andygrove)

  • chore(deps): upgrade to DataFusion 54 #1906 (andygrove)

  • ci: verify TPC-H query results against single-process DataFusion #1917 (andygrove)

  • chore(deps): upgrade pyballista to DataFusion 54 #1919 (djanand)

  • Fix/python cargo lock drift #1899 (djanand)

  • refactor(scheduler): keep submit_job non-breaking, add submit_physical_plan #1941 (andygrove)

  • Add TPC-H distributed plan-stability test suite #1966 (andygrove)

  • Display input stage id on UnresolvedShuffleExec #1969 (andygrove)

  • chore(deps): bump pyarrow from 23.0.0 to 23.0.1 in /python #1961 (dependabot[bot])

  • chore: add script and CI check to sync vendored DataFusion protos #1991 (andygrove)

  • feat(tui + rest): surface failed tasks #1949 (killzoner)

  • Add shuffle-read fetch metrics and surface per-operator metrics in plan display #1968 (andygrove)

  • Log task execution duration in the executor “Finished task” message #1999 (andygrove)

  • chore: bump Ballista version to 54.0.0 #2004 (andygrove)

Credits#

Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.

    83	dependabot[bot]
    38	Andy Grove
    18	Martin Grigorov
    14	Marko Milenković
     4	Bhargava Vadlamani
     3	Jarro van Ginkel
     3	alexander domenti
     3	jgrim
     2	Gunther Xing
     2	Jogesh A Dinavahi
     1	Jogesh Dinavahi

Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.