<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->

# Apache DataFusion Ballista 54.0.0 Changelog

This release consists of 171 commits from 11 contributors. See credits at the end of this changelog for more information.

**Fixed bugs:**

- fix(python): clear latent lint debt and gate wheel jobs on lint/tests [#1749](https://github.com/apache/datafusion-ballista/pull/1749) (andygrove)
- fix(python): exclude mimalloc from Python wheel to avoid macOS segfault [#1769](https://github.com/apache/datafusion-ballista/pull/1769) (andygrove)
- fix: handle task execution errors and panics in executor [#1793](https://github.com/apache/datafusion-ballista/pull/1793) (milenkovicm)
- fix: downgrade `time` version to fix build issue [#1862](https://github.com/apache/datafusion-ballista/pull/1862) (milenkovicm)
- fix: use DataFrame.**name** instead of hardcoded string in metaclass method wrapping [#1866](https://github.com/apache/datafusion-ballista/pull/1866) (djanand)
- fix: honor client datafusion.\* session config overrides in scheduler planning [#1902](https://github.com/apache/datafusion-ballista/pull/1902) (andygrove)
- fix: Adaptive query planning fails with remote object stores: "No suitable object store found" [#1962](https://github.com/apache/datafusion-ballista/pull/1962) (milenkovicm)
- fix: satisfy clippy 1.97 useless_borrows_in_formatting [#1978](https://github.com/apache/datafusion-ballista/pull/1978) (andygrove)

**Performance related:**

- perf: avoid redundant TaskStatus clones in scheduler update_task_status [#1983](https://github.com/apache/datafusion-ballista/pull/1983) (andygrove)
- perf: prune irrelevant partitions task [#1911](https://github.com/apache/datafusion-ballista/pull/1911) (coderfender)

**Implemented enhancements:**

- feat: task_manager delegats physical plan creation to execution graph [#1726](https://github.com/apache/datafusion-ballista/pull/1726) (milenkovicm)
- feat: ShuffleReader shows `upstream_stage` [#1767](https://github.com/apache/datafusion-ballista/pull/1767) (milenkovicm)
- feat: runtime stage's metrics rendering [#1778](https://github.com/apache/datafusion-ballista/pull/1778) (sandugood)
- feat: add `/api/job/{job_id}/config` endpoint to retrieve session configuration [#1792](https://github.com/apache/datafusion-ballista/pull/1792) (killzoner)
- feat(aqe): delay join decision & introduce broadcast join in AQE [#1752](https://github.com/apache/datafusion-ballista/pull/1752) (milenkovicm)
- feat: Add web rendering of the TUI application [#1742](https://github.com/apache/datafusion-ballista/pull/1742) (martin-g)
- feat(aqe): chaos monkey testing for ballista execution [#1802](https://github.com/apache/datafusion-ballista/pull/1802) (milenkovicm)
- feat(aqe): empty join handling [#1723](https://github.com/apache/datafusion-ballista/pull/1723) (sandugood)
- feat(tui): Show shimmer animation for "Queued" and "Running" jobs [#1808](https://github.com/apache/datafusion-ballista/pull/1808) (martin-g)
- feat(cli): use local time in tracing subscriber logging for ballista-cli [#1824](https://github.com/apache/datafusion-ballista/pull/1824) (jiengup)
- feat(scheduler): Use command line options for CORS origins/methods instead of env vars [#1818](https://github.com/apache/datafusion-ballista/pull/1818) (martin-g)
- feat(tui): Add configuration popup for jobs [#1825](https://github.com/apache/datafusion-ballista/pull/1825) (jiengup)
- feat(TUI): enable various plan rendering formats [#1828](https://github.com/apache/datafusion-ballista/pull/1828) (sandugood)
- feat(tui): Add theming support [#1850](https://github.com/apache/datafusion-ballista/pull/1850) (martin-g)
- feat: NewType pattern for `job_id` [#1852](https://github.com/apache/datafusion-ballista/pull/1852) (JarroVGIT)
- feat(tui): Show job failed status below the Jobs table [#1871](https://github.com/apache/datafusion-ballista/pull/1871) (martin-g)
- feat: broadcast small build sides via non-zero CollectLeft thresholds [#1900](https://github.com/apache/datafusion-ballista/pull/1900) (andygrove)
- feat: broadcast small build side of SortMergeJoinExec in the static planner [#1904](https://github.com/apache/datafusion-ballista/pull/1904) (andygrove)
- feat: Allow submitting a pre-built physical plan to the scheduler [#1924](https://github.com/apache/datafusion-ballista/pull/1924) (andygrove)
- feat: cancel running stages and tasks on job failure or cancellation [#1903](https://github.com/apache/datafusion-ballista/pull/1903) (killzoner)
- feat: support running TPC-H benchmarks against S3-compatible object storage [#1942](https://github.com/apache/datafusion-ballista/pull/1942) (andygrove)
- feat: bound shuffle fetch with a reduce-side in-flight governor [#1951](https://github.com/apache/datafusion-ballista/pull/1951) (andygrove)
- feat: improve session state creation for AQE planner [#1963](https://github.com/apache/datafusion-ballista/pull/1963) (milenkovicm)
- feat: delete intermediate shuffle files on job success [#1982](https://github.com/apache/datafusion-ballista/pull/1982) (andygrove)
- feat: Share read-side runtime state across a session's tasks on the executor [#1995](https://github.com/apache/datafusion-ballista/pull/1995) (andygrove)

**Documentation updates:**

- docs(release): document PyPI publish process for python client [#1610](https://github.com/apache/datafusion-ballista/pull/1610) (andygrove)
- docs(release): cover TestPyPI verification and fix stale wheel-download notes [#1760](https://github.com/apache/datafusion-ballista/pull/1760) (andygrove)
- refactor: make use of `async_trait::async_trait` consistent across project [#1848](https://github.com/apache/datafusion-ballista/pull/1848) (JarroVGIT)
- chore(docs): Update docs website to bring inline with Datafusion style [#1857](https://github.com/apache/datafusion-ballista/pull/1857) (JarroVGIT)
- Improve changelog generator and split CHANGELOG.md into per-version files [#1768](https://github.com/apache/datafusion-ballista/pull/1768) (andygrove)
- docs: add versioned upgrade guide (54.0.0) [#1939](https://github.com/apache/datafusion-ballista/pull/1939) (andygrove)
- docs: document TPC-H plan-stability golden files and regeneration [#1979](https://github.com/apache/datafusion-ballista/pull/1979) (andygrove)
- docs: update companies using ballista [#1992](https://github.com/apache/datafusion-ballista/pull/1992) (coderfender)
- docs: link AQE limitations to tracking issues and add AQE contributor guide section [#1990](https://github.com/apache/datafusion-ballista/pull/1990) (andygrove)

**Other:**

- chore(deps): bump idna from 3.11 to 3.15 in /python [#1737](https://github.com/apache/datafusion-ballista/pull/1737) (dependabot[bot])
- chore(ci): Execute the CI workflows only when this is a push to main or a PR against any branch [#1738](https://github.com/apache/datafusion-ballista/pull/1738) (martin-g)
- minor: move aqe executions plans to a directory [#1744](https://github.com/apache/datafusion-ballista/pull/1744) (milenkovicm)
- ci: pin all third-party actions to ASF-allowlisted SHAs [#1748](https://github.com/apache/datafusion-ballista/pull/1748) (andygrove)
- ci: pin Windows wheel builder to windows-2022 [#1751](https://github.com/apache/datafusion-ballista/pull/1751) (andygrove)
- chore: bump python crate to 53.0.0 and use path deps for ballista crates [#1750](https://github.com/apache/datafusion-ballista/pull/1750) (andygrove)
- ci: produce real aarch64 wheel and upload sdist artifact [#1753](https://github.com/apache/datafusion-ballista/pull/1753) (andygrove)
- ci: only run Python Release Build on RC tags and matching PRs [#1754](https://github.com/apache/datafusion-ballista/pull/1754) (andygrove)
- ci: gate TPC-H workflow on rust-only path allow list [#1755](https://github.com/apache/datafusion-ballista/pull/1755) (andygrove)
- ci: gate Docker workflow on rust-only path allow list [#1756](https://github.com/apache/datafusion-ballista/pull/1756) (andygrove)
- ci: gate Rust workflow on workspace path allow list and rename to Ballista Rust [#1757](https://github.com/apache/datafusion-ballista/pull/1757) (andygrove)
- ci: use maturin sdist subcommand to avoid building a wheel [#1759](https://github.com/apache/datafusion-ballista/pull/1759) (andygrove)
- ci: gate CodeQL workflow on .github path changes [#1758](https://github.com/apache/datafusion-ballista/pull/1758) (andygrove)
- minor: Ballista `showplan` helper script [#1761](https://github.com/apache/datafusion-ballista/pull/1761) (milenkovicm)
- chore(deps): bump requests from 2.32.5 to 2.33.0 in /python [#1781](https://github.com/apache/datafusion-ballista/pull/1781) (dependabot[bot])
- chore(ci): Execute the "Dependencies" workflow on push to main and for PRs [#1782](https://github.com/apache/datafusion-ballista/pull/1782) (martin-g)
- chore(deps): bump astral-tokio-tar from 0.6.1 to 0.6.2 [#1794](https://github.com/apache/datafusion-ballista/pull/1794) (dependabot[bot])
- chore(deps): bump pygments from 2.19.2 to 2.20.0 in /python [#1800](https://github.com/apache/datafusion-ballista/pull/1800) (dependabot[bot])
- minor: code cleanup and log level changes [#1805](https://github.com/apache/datafusion-ballista/pull/1805) (milenkovicm)
- minor(aqe): backport few datafusion physical rules identified as not idempotent [#1806](https://github.com/apache/datafusion-ballista/pull/1806) (milenkovicm)
- minor(aqe): add `plan_id` to dynamic join [#1809](https://github.com/apache/datafusion-ballista/pull/1809) (milenkovicm)
- chore(docker): Use yellow_tripdata_2022-01.parquet from nightlies.apache.org [#1817](https://github.com/apache/datafusion-ballista/pull/1817) (martin-g)
- chore(ci): Use MacOS Github hosted runners for the small CI checks [#1810](https://github.com/apache/datafusion-ballista/pull/1810) (martin-g)
- chore(ci): Fix the Rust installer for MacOS in dependencies.yml [#1833](https://github.com/apache/datafusion-ballista/pull/1833) (martin-g)
- chore(ci): Drop Windows CI runs [#1801](https://github.com/apache/datafusion-ballista/pull/1801) (martin-g)
- chore(ci): Fix the name of a CI job dependency [#1834](https://github.com/apache/datafusion-ballista/pull/1834) (martin-g)
- chore(github): Enable "Update branch" button for PRs [#1839](https://github.com/apache/datafusion-ballista/pull/1839) (martin-g)
- chore(ci): Download macos aarch64 build of protoc [#1835](https://github.com/apache/datafusion-ballista/pull/1835) (martin-g)
- chore(tui): Upgrade Ratatui to 0.30.1 / Ratzilla to 0.3.1 [#1840](https://github.com/apache/datafusion-ballista/pull/1840) (martin-g)
- chore(ci): Run `cargo check` and `cargo doc` CI jobs on MacOS [#1846](https://github.com/apache/datafusion-ballista/pull/1846) (martin-g)
- chore: re enable approx distinct test [#1844](https://github.com/apache/datafusion-ballista/pull/1844) (coderfender)
- bug: fix empty projection serde ambiguity bug [#1863](https://github.com/apache/datafusion-ballista/pull/1863) (coderfender)
- chore(tui): Use lighter foreground colors for the dark theme. For better contrast [#1872](https://github.com/apache/datafusion-ballista/pull/1872) (martin-g)
- minor: remove redundant scheduler info loggers [#1879](https://github.com/apache/datafusion-ballista/pull/1879) (milenkovicm)
- chore(ci): Deploy WebTUI to nightlies.a.o [#1888](https://github.com/apache/datafusion-ballista/pull/1888) (martin-g)
- ci: run TPC-H SF10 under AQE off + on, and build it faster [#1913](https://github.com/apache/datafusion-ballista/pull/1913) (andygrove)
- chore(deps): upgrade to DataFusion 54 [#1906](https://github.com/apache/datafusion-ballista/pull/1906) (andygrove)
- ci: verify TPC-H query results against single-process DataFusion [#1917](https://github.com/apache/datafusion-ballista/pull/1917) (andygrove)
- chore(deps): upgrade pyballista to DataFusion 54 [#1919](https://github.com/apache/datafusion-ballista/pull/1919) (djanand)
- Fix/python cargo lock drift [#1899](https://github.com/apache/datafusion-ballista/pull/1899) (djanand)
- refactor(scheduler): keep submit_job non-breaking, add submit_physical_plan [#1941](https://github.com/apache/datafusion-ballista/pull/1941) (andygrove)
- Add TPC-H distributed plan-stability test suite [#1966](https://github.com/apache/datafusion-ballista/pull/1966) (andygrove)
- Display input stage id on UnresolvedShuffleExec [#1969](https://github.com/apache/datafusion-ballista/pull/1969) (andygrove)
- chore(deps): bump pyarrow from 23.0.0 to 23.0.1 in /python [#1961](https://github.com/apache/datafusion-ballista/pull/1961) (dependabot[bot])
- chore: add script and CI check to sync vendored DataFusion protos [#1991](https://github.com/apache/datafusion-ballista/pull/1991) (andygrove)
- feat(tui + rest): surface failed tasks [#1949](https://github.com/apache/datafusion-ballista/pull/1949) (killzoner)
- Add shuffle-read fetch metrics and surface per-operator metrics in plan display [#1968](https://github.com/apache/datafusion-ballista/pull/1968) (andygrove)
- Log task execution duration in the executor "Finished task" message [#1999](https://github.com/apache/datafusion-ballista/pull/1999) (andygrove)
- chore: bump Ballista version to 54.0.0 [#2004](https://github.com/apache/datafusion-ballista/pull/2004) (andygrove)

## Credits

Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.

```
    83	dependabot[bot]
    38	Andy Grove
    18	Martin Grigorov
    14	Marko Milenković
     4	Bhargava Vadlamani
     3	Jarro van Ginkel
     3	alexander domenti
     3	jgrim
     2	Gunther Xing
     2	Jogesh A Dinavahi
     1	Jogesh Dinavahi
```

Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.