<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->

# Apache DataFusion Ballista 53.0.0 Changelog

This release consists of 174 commits from 16 contributors. See credits at the end of this changelog for more information.

**Fixed bugs:**

- fix: remove unwrap from executor and improve task error handling [#1540](https://github.com/apache/datafusion-ballista/pull/1540) (eachsaj)
- fix: handle None task slot in update_task_info after executor lost [#1523](https://github.com/apache/datafusion-ballista/pull/1523) (milenkovicm)
- fix: [Python] serialize optimized logical plan to fix subquery support [#1586](https://github.com/apache/datafusion-ballista/pull/1586) (andygrove)
- fix: allow S3 access without explicit credentials [#1584](https://github.com/apache/datafusion-ballista/pull/1584) (andygrove)
- fix: route collect/show/to_pandas through Ballista cluster [#1585](https://github.com/apache/datafusion-ballista/pull/1585) (andygrove)
- fix: propagate session config from Python client to Ballista cluster [#1592](https://github.com/apache/datafusion-ballista/pull/1592) (andygrove)
- fix(python): ignore ballista-namespaced cluster_config keys locally [#1613](https://github.com/apache/datafusion-ballista/pull/1613) (andygrove)
- fix: `df.write_` fix as it was broken after update [#1625](https://github.com/apache/datafusion-ballista/pull/1625) (milenkovicm)
- fix(rest): remove unwrap and return 404 if executor does not exist [#1628](https://github.com/apache/datafusion-ballista/pull/1628) (milenkovicm)
- fix(metrics): avoid stage metrics inflation by tracking partition snapshots [#1652](https://github.com/apache/datafusion-ballista/pull/1652) (danielhumanmod)
- fix: compilation issue after merge [#1658](https://github.com/apache/datafusion-ballista/pull/1658) (milenkovicm)
- fix: rest api calculates stage running time correctly [#1675](https://github.com/apache/datafusion-ballista/pull/1675) (milenkovicm)
- fix: REST API does not show running jobs [#1703](https://github.com/apache/datafusion-ballista/pull/1703) (gittihub-jpg)
- fix: no executor warning, correct prometheus feature name in TUI [#1698](https://github.com/apache/datafusion-ballista/pull/1698) (killzoner)

**Implemented enhancements:**

- [REST]: Cancelling a completed/failed job should return "false" [#1494](https://github.com/apache/datafusion-ballista/pull/1494) (martin-g)
- feat: Add plain "status" field to the JobResponse [#1497](https://github.com/apache/datafusion-ballista/pull/1497) (martin-g)
- feat: Expose Logical and Physical plan details in the REST API [#1498](https://github.com/apache/datafusion-ballista/pull/1498) (milenkovicm)
- feat: (remote) shuffle reader cleanup [#1503](https://github.com/apache/datafusion-ballista/pull/1503) (milenkovicm)
- feat: enable scheduler rest api by default [#1506](https://github.com/apache/datafusion-ballista/pull/1506) (milenkovicm)
- feat: [REST] Return the job's start and end times [#1511](https://github.com/apache/datafusion-ballista/pull/1511) (martin-g)
- feat: remove hash policy task distribution as not used [#1526](https://github.com/apache/datafusion-ballista/pull/1526) (eachsaj)
- feat: remove full path from partition locations [#1527](https://github.com/apache/datafusion-ballista/pull/1527) (milenkovicm)
- feat: add config to ExecutorEngine::create_query_stage_exec [#1542](https://github.com/apache/datafusion-ballista/pull/1542) (milenkovicm)
- feat: Improve REST API adding task related info to job stages [#1543](https://github.com/apache/datafusion-ballista/pull/1543) (milenkovicm)
- feat: Ballista Text User Interface [#1436](https://github.com/apache/datafusion-ballista/pull/1436) (martin-g)
- feat: jupyter notebook support [#1513](https://github.com/apache/datafusion-ballista/pull/1513) (sandugood)
- feat: ExecutionEngine::create_query_stage_exec accepts partition_id [#1556](https://github.com/apache/datafusion-ballista/pull/1556) (milenkovicm)
- feat: make ballista client retry policy configurable [#1577](https://github.com/apache/datafusion-ballista/pull/1577) (eachsaj)
- feat: Executor system & process metrics reporting [#1547](https://github.com/apache/datafusion-ballista/pull/1547) (sandugood)
- feat: Scheduler config update [#1597](https://github.com/apache/datafusion-ballista/pull/1597) (sandugood)
- feat: Support `EXPLAIN ANALYZE` in Ballista [#1567](https://github.com/apache/datafusion-ballista/pull/1567) (danielhumanmod)
- feat: Add standalone shuffle writer benchmark that shuffles real Parquet input [#1600](https://github.com/apache/datafusion-ballista/pull/1600) (andygrove)
- feat: defer sort-shuffle materialization with interleave_record_batch [#1598](https://github.com/apache/datafusion-ballista/pull/1598) (andygrove)
- feat(executor): bound executor memory via --memory-pool-size [#1624](https://github.com/apache/datafusion-ballista/pull/1624) (andygrove)
- feat(bench): show TPC-H query timings in seconds and add total time [#1641](https://github.com/apache/datafusion-ballista/pull/1641) (andygrove)
- feat(aqe): Support sort-based shuffle writer in AQE [#1640](https://github.com/apache/datafusion-ballista/pull/1640) (danielhumanmod)
- feat: rest api supports plan tree rendering [#1650](https://github.com/apache/datafusion-ballista/pull/1650) (sandugood)
- feat: Cache ballista clients on executor [#1578](https://github.com/apache/datafusion-ballista/pull/1578) (milenkovicm)
- feat(aqe): Lazy stage evaluation in AQE [#1649](https://github.com/apache/datafusion-ballista/pull/1649) (milenkovicm)
- feat: move shuffle writer disk I/O off tokio worker threads [#1537](https://github.com/apache/datafusion-ballista/pull/1537) (hcrosse)
- feat(scheduler): broadcast-style hash join for small-side joins [#1647](https://github.com/apache/datafusion-ballista/pull/1647) (andygrove)
- feat: default to sort-merge join [#1651](https://github.com/apache/datafusion-ballista/pull/1651) (andygrove)
- feat: TUI shows running job information [#1717](https://github.com/apache/datafusion-ballista/pull/1717) (milenkovicm)
- feat(aqe): CoalescePartitionsRule — shuffle-partition coalescing on resolved stats [#1684](https://github.com/apache/datafusion-ballista/pull/1684) (metegenez)
- feat: TUI make task popup scrollable [#1725](https://github.com/apache/datafusion-ballista/pull/1725) (milenkovicm)
- feat(tui): Use separate areas for the table and its associated scrollbar [#1729](https://github.com/apache/datafusion-ballista/pull/1729) (martin-g)

**Documentation updates:**

- docs: document hash-based and sort-based shuffle implementations [#1595](https://github.com/apache/datafusion-ballista/pull/1595) (andygrove)
- docs: improve Python documentation structure [#1579](https://github.com/apache/datafusion-ballista/pull/1579) (andygrove)
- docs: add Python client build-from-source section to contributor guide [#1620](https://github.com/apache/datafusion-ballista/pull/1620) (andygrove)
- chore: remove NYC Taxi benchmark [#1644](https://github.com/apache/datafusion-ballista/pull/1644) (andygrove)
- docs: document experimental Adaptive Query Execution in tuning guide [#1645](https://github.com/apache/datafusion-ballista/pull/1645) (andygrove)
- build(bench): rework docker-compose TPC-H stack [#1646](https://github.com/apache/datafusion-ballista/pull/1646) (andygrove)
- docs: add Ballista TUI documentation [#1593](https://github.com/apache/datafusion-ballista/pull/1593) (goingforstudying-ctrl)
- [TUI] Show executor's details in a popup [#1670](https://github.com/apache/datafusion-ballista/pull/1670) (martin-g)
- [TUI] Add a config setting for rendering job stage's plan as a tree [#1704](https://github.com/apache/datafusion-ballista/pull/1704) (martin-g)
- [TUI] Add support for horizontal scrolling to the job/stage plan popups [#1711](https://github.com/apache/datafusion-ballista/pull/1711) (martin-g)
- [TUI] Add screenshots of the TUI application in README/cli.md [#1714](https://github.com/apache/datafusion-ballista/pull/1714) (martin-g)

**Other:**

- CI: Add CodeQL workflow for GitHub Actions security scanning [#1484](https://github.com/apache/datafusion-ballista/pull/1484) (kevinjqliu)
- ci: Harden labeler workflow, remove unnecessary checkout from pull_request_target job [#1487](https://github.com/apache/datafusion-ballista/pull/1487) (kevinjqliu)
- chore(deps): bump tokio from 1.49.0 to 1.50.0 [#1490](https://github.com/apache/datafusion-ballista/pull/1490) (dependabot[bot])
- chore(deps): bump actions/setup-node from 6.2.0 to 6.3.0 [#1489](https://github.com/apache/datafusion-ballista/pull/1489) (dependabot[bot])
- ci: add take and stale [#1488](https://github.com/apache/datafusion-ballista/pull/1488) (kevinjqliu)
- chore(deps): bump uuid from 1.21.0 to 1.22.0 [#1492](https://github.com/apache/datafusion-ballista/pull/1492) (dependabot[bot])
- chore(deps): bump github/codeql-action from 4.32.5 to 4.32.6 [#1491](https://github.com/apache/datafusion-ballista/pull/1491) (dependabot[bot])
- chore(deps): bump libc from 0.2.182 to 0.2.183 [#1495](https://github.com/apache/datafusion-ballista/pull/1495) (dependabot[bot])
- chore(deps): bump graphviz-rust from 0.9.6 to 0.9.7 [#1496](https://github.com/apache/datafusion-ballista/pull/1496) (dependabot[bot])
- chore(deps): bump quinn-proto from 0.11.13 to 0.11.14 [#1499](https://github.com/apache/datafusion-ballista/pull/1499) (dependabot[bot])
- chore(deps): bump quinn-proto from 0.11.13 to 0.11.14 in /python [#1500](https://github.com/apache/datafusion-ballista/pull/1500) (dependabot[bot])
- chore(deps): bump tempfile from 3.26.0 to 3.27.0 [#1501](https://github.com/apache/datafusion-ballista/pull/1501) (dependabot[bot])
- chore(deps): cargo update deps [#1502](https://github.com/apache/datafusion-ballista/pull/1502) (milenkovicm)
- chore(deps): bump once_cell from 1.21.3 to 1.21.4 [#1505](https://github.com/apache/datafusion-ballista/pull/1505) (dependabot[bot])
- chore(deps): bump clap from 4.5.60 to 4.6.0 [#1504](https://github.com/apache/datafusion-ballista/pull/1504) (dependabot[bot])
- minor: task scheduling policy config cleanup [#1507](https://github.com/apache/datafusion-ballista/pull/1507) (milenkovicm)
- minor: address task policy config comments [#1508](https://github.com/apache/datafusion-ballista/pull/1508) (milenkovicm)
- chore(deps): bump tracing-subscriber from 0.3.22 to 0.3.23 [#1510](https://github.com/apache/datafusion-ballista/pull/1510) (dependabot[bot])
- minor: [REST] add datafusion version info [#1512](https://github.com/apache/datafusion-ballista/pull/1512) (milenkovicm)
- chore(deps): bump lz4_flex from 0.12.0 to 0.12.1 in /python [#1514](https://github.com/apache/datafusion-ballista/pull/1514) (dependabot[bot])
- chore(deps): bump github/codeql-action from 4.32.6 to 4.33.0 [#1515](https://github.com/apache/datafusion-ballista/pull/1515) (dependabot[bot])
- chore(deps): bump rustls-webpki from 0.103.9 to 0.103.10 in /python [#1517](https://github.com/apache/datafusion-ballista/pull/1517) (dependabot[bot])
- chore(deps): bump lz4_flex from 0.12.0 to 0.12.1 [#1518](https://github.com/apache/datafusion-ballista/pull/1518) (dependabot[bot])
- chore(deps): bump github/codeql-action from 4.33.0 to 4.34.1 [#1519](https://github.com/apache/datafusion-ballista/pull/1519) (dependabot[bot])
- chore(deps): bump rustls-webpki from 0.103.9 to 0.103.10 [#1521](https://github.com/apache/datafusion-ballista/pull/1521) (dependabot[bot])
- ci: pin third-party actions to Apache-approved SHAs [#1516](https://github.com/apache/datafusion-ballista/pull/1516) (kevinjqliu)
- chore(deps): bump env_logger from 0.11.9 to 0.11.10 [#1522](https://github.com/apache/datafusion-ballista/pull/1522) (dependabot[bot])
- chore(deps): update to datafusion v.53 [#1486](https://github.com/apache/datafusion-ballista/pull/1486) (milenkovicm)
- chore(deps): bump insta from 1.46.3 to 1.47.0 [#1524](https://github.com/apache/datafusion-ballista/pull/1524) (dependabot[bot])
- chore(deps): bump uuid from 1.22.0 to 1.23.0 [#1525](https://github.com/apache/datafusion-ballista/pull/1525) (dependabot[bot])
- chore: update datafusion proto [#1528](https://github.com/apache/datafusion-ballista/pull/1528) (milenkovicm)
- minor: cleanup scheduler clap configuration [#1529](https://github.com/apache/datafusion-ballista/pull/1529) (milenkovicm)
- chore(deps): bump insta from 1.47.0 to 1.47.1 [#1532](https://github.com/apache/datafusion-ballista/pull/1532) (dependabot[bot])
- chore(deps): bump ctor from 0.6.3 to 0.8.0 [#1534](https://github.com/apache/datafusion-ballista/pull/1534) (dependabot[bot])
- chore(deps): bump md-5 from 0.10.6 to 0.11.0 [#1533](https://github.com/apache/datafusion-ballista/pull/1533) (dependabot[bot])
- chore(deps): bump github/codeql-action from 4.34.1 to 4.35.1 [#1530](https://github.com/apache/datafusion-ballista/pull/1530) (dependabot[bot])
- chore(deps): bump insta from 1.47.1 to 1.47.2 [#1535](https://github.com/apache/datafusion-ballista/pull/1535) (dependabot[bot])
- chore(deps): bump libc from 0.2.183 to 0.2.184 [#1538](https://github.com/apache/datafusion-ballista/pull/1538) (dependabot[bot])
- chore(deps): bump tokio from 1.50.0 to 1.51.0 [#1544](https://github.com/apache/datafusion-ballista/pull/1544) (dependabot[bot])
- chore(deps): bump tui-big-text from 0.8.3 to 0.8.4 [#1546](https://github.com/apache/datafusion-ballista/pull/1546) (dependabot[bot])
- chore(deps): bump tokio from 1.51.0 to 1.51.1 [#1545](https://github.com/apache/datafusion-ballista/pull/1545) (dependabot[bot])
- chore(deps): bump ctor from 0.8.0 to 0.9.1 [#1548](https://github.com/apache/datafusion-ballista/pull/1548) (dependabot[bot])
- chore(deps): bump ctor from 0.9.1 to 0.10.0 [#1549](https://github.com/apache/datafusion-ballista/pull/1549) (dependabot[bot])
- chore(deps): bump rustls from 0.23.37 to 0.23.38 [#1550](https://github.com/apache/datafusion-ballista/pull/1550) (dependabot[bot])
- chore(deps): bump libc from 0.2.184 to 0.2.185 [#1553](https://github.com/apache/datafusion-ballista/pull/1553) (dependabot[bot])
- chore(deps): bump rand from 0.9.2 to 0.9.4 in /python [#1552](https://github.com/apache/datafusion-ballista/pull/1552) (dependabot[bot])
- chore(deps): bump tokio from 1.51.1 to 1.52.0 [#1555](https://github.com/apache/datafusion-ballista/pull/1555) (dependabot[bot])
- chore(deps): bump axum from 0.8.8 to 0.8.9 [#1554](https://github.com/apache/datafusion-ballista/pull/1554) (dependabot[bot])
- chore(deps): bump github/codeql-action from 4.35.1 to 4.35.2 [#1557](https://github.com/apache/datafusion-ballista/pull/1557) (dependabot[bot])
- chore: Fix Clippy issues with Rust 1.95.0 [#1558](https://github.com/apache/datafusion-ballista/pull/1558) (martin-g)
- chore: update deps [#1561](https://github.com/apache/datafusion-ballista/pull/1561) (milenkovicm)
- chore(deps): bump actions/setup-node from 6.3.0 to 6.4.0 [#1562](https://github.com/apache/datafusion-ballista/pull/1562) (dependabot[bot])
- chore(deps): bump mimalloc from 0.1.48 to 0.1.49 [#1566](https://github.com/apache/datafusion-ballista/pull/1566) (dependabot[bot])
- chore(deps): bump aws-config from 1.8.15 to 1.8.16 [#1564](https://github.com/apache/datafusion-ballista/pull/1564) (dependabot[bot])
- chore(deps): bump rand from 0.9.4 to 0.10.1 [#1565](https://github.com/apache/datafusion-ballista/pull/1565) (dependabot[bot])
- chore(deps): bump ctor from 0.10.0 to 0.10.1 [#1571](https://github.com/apache/datafusion-ballista/pull/1571) (dependabot[bot])
- chore(deps): bump mimalloc from 0.1.49 to 0.1.50 [#1570](https://github.com/apache/datafusion-ballista/pull/1570) (dependabot[bot])
- chore(deps): bump rustls from 0.23.38 to 0.23.39 [#1569](https://github.com/apache/datafusion-ballista/pull/1569) (dependabot[bot])
- chore(deps): bump libc from 0.2.185 to 0.2.186 [#1572](https://github.com/apache/datafusion-ballista/pull/1572) (dependabot[bot])
- [TUI] Change the key binding for job's plans [#1573](https://github.com/apache/datafusion-ballista/pull/1573) (martin-g)
- chore(deps): bump rustls-webpki from 0.103.12 to 0.103.13 [#1576](https://github.com/apache/datafusion-ballista/pull/1576) (dependabot[bot])
- chore(deps): bump rustls-webpki from 0.103.10 to 0.103.13 in /python [#1575](https://github.com/apache/datafusion-ballista/pull/1575) (dependabot[bot])
- minor: make config naming convention consistent [#1580](https://github.com/apache/datafusion-ballista/pull/1580) (milenkovicm)
- chore: Remove unnecessary optimizer rules due to datafusion upgrade to v53 [#1594](https://github.com/apache/datafusion-ballista/pull/1594) (sandugood)
- chore(deps): bump reqwest from 0.13.2 to 0.13.3 [#1605](https://github.com/apache/datafusion-ballista/pull/1605) (dependabot[bot])
- ci: drop Intel macOS Python wheel build [#1612](https://github.com/apache/datafusion-ballista/pull/1612) (andygrove)
- chore(deps): bump ctor from 0.10.1 to 0.11.1 [#1618](https://github.com/apache/datafusion-ballista/pull/1618) (dependabot[bot])
- chore(deps): bump rustls from 0.23.39 to 0.23.40 [#1619](https://github.com/apache/datafusion-ballista/pull/1619) (dependabot[bot])
- chore: update python deps to ballista and datafusion 52 [#1590](https://github.com/apache/datafusion-ballista/pull/1590) (andygrove)
- feat(sort-shuffle): byte-copy spill files and enable block-IO transport [#1615](https://github.com/apache/datafusion-ballista/pull/1615) (andygrove)
- Use BallistaSessionContext instead of BallistaBuilder in tpch.py [#1621](https://github.com/apache/datafusion-ballista/pull/1621) (martin-g)
- feat(sort-shuffle): enable sort-based shuffle by default [#1623](https://github.com/apache/datafusion-ballista/pull/1623) (andygrove)
- perf(sort-shuffle): fix performance regression caused by datafusion upgrade [#1626](https://github.com/apache/datafusion-ballista/pull/1626) (andygrove)
- chore(deps): bump ctor from 0.11.1 to 0.12.0 [#1639](https://github.com/apache/datafusion-ballista/pull/1639) (dependabot[bot])
- fix(sort-shuffle): bound writer memory with per-task spill threshold [#1636](https://github.com/apache/datafusion-ballista/pull/1636) (andygrove)
- chore(deps): bump graphviz-rust from 0.9.7 to 0.9.8 [#1655](https://github.com/apache/datafusion-ballista/pull/1655) (dependabot[bot])
- chore(deps): bump github/codeql-action from 4.35.2 to 4.35.3 [#1653](https://github.com/apache/datafusion-ballista/pull/1653) (dependabot[bot])
- chore(deps): bump tokio from 1.52.1 to 1.52.2 [#1657](https://github.com/apache/datafusion-ballista/pull/1657) (dependabot[bot])
- chore(deps): bump actions/labeler from 6.0.1 to 6.1.0 [#1659](https://github.com/apache/datafusion-ballista/pull/1659) (dependabot[bot])
- [TUI] Show job's stages and their tasks [#1574](https://github.com/apache/datafusion-ballista/pull/1574) (martin-g)
- chore(deps): bump tonic-build from 0.14.5 to 0.14.6 [#1662](https://github.com/apache/datafusion-ballista/pull/1662) (dependabot[bot])
- chore(deps): bump tonic from 0.14.5 to 0.14.6 [#1661](https://github.com/apache/datafusion-ballista/pull/1661) (dependabot[bot])
- chore(deps): bump astral-tokio-tar from 0.6.0 to 0.6.1 [#1664](https://github.com/apache/datafusion-ballista/pull/1664) (dependabot[bot])
- chore(deps): bump ctor from 0.12.0 to 1.0.2 [#1663](https://github.com/apache/datafusion-ballista/pull/1663) (dependabot[bot])
- chore(deps): bump github/codeql-action from 4.35.3 to 4.35.4 [#1665](https://github.com/apache/datafusion-ballista/pull/1665) (dependabot[bot])
- chore(deps): bump ctor from 1.0.2 to 1.0.3 [#1668](https://github.com/apache/datafusion-ballista/pull/1668) (dependabot[bot])
- chore(deps): bump tonic-prost from 0.14.5 to 0.14.6 [#1667](https://github.com/apache/datafusion-ballista/pull/1667) (dependabot[bot])
- chore(deps): bump tonic-prost-build from 0.14.5 to 0.14.6 [#1666](https://github.com/apache/datafusion-ballista/pull/1666) (dependabot[bot])
- [TUI] Configurable tick interval [#1669](https://github.com/apache/datafusion-ballista/pull/1669) (martin-g)
- minor: add pending stage indicator for `AdaptiveDatafusionExec` [#1672](https://github.com/apache/datafusion-ballista/pull/1672) (milenkovicm)
- chore(deps): bump ctor from 1.0.3 to 1.0.4 [#1680](https://github.com/apache/datafusion-ballista/pull/1680) (dependabot[bot])
- chore(deps): bump tokio from 1.52.2 to 1.52.3 [#1678](https://github.com/apache/datafusion-ballista/pull/1678) (dependabot[bot])
- minor: change parameter ordering in `AdaptivePlanner::try_new_with_optimizers` [#1687](https://github.com/apache/datafusion-ballista/pull/1687) (milenkovicm)
- chore(deps): bump ctor from 1.0.4 to 1.0.5 [#1694](https://github.com/apache/datafusion-ballista/pull/1694) (dependabot[bot])
- Fix REST API panic on job list/detail when end_time < start_time [#1693](https://github.com/apache/datafusion-ballista/pull/1693) (abhinavgautam01)
- Right align all numeric columns in the TUI tables [#1695](https://github.com/apache/datafusion-ballista/pull/1695) (martin-g)
- fix(join-selection): guard CollectLeft swap when right has multiple partitions [#1691](https://github.com/apache/datafusion-ballista/pull/1691) (andygrove)
- Minor improvements for #1675 [#1686](https://github.com/apache/datafusion-ballista/pull/1686) (martin-g)
- [CLI/TUI] Use only tracing crate for logging in CLI and TUI [#1697](https://github.com/apache/datafusion-ballista/pull/1697) (martin-g)
- chore(deps): bump urllib3 from 2.6.3 to 2.7.0 in /python [#1699](https://github.com/apache/datafusion-ballista/pull/1699) (dependabot[bot])
- chore(deps): bump pyjwt from 2.10.1 to 2.12.0 in /python [#1700](https://github.com/apache/datafusion-ballista/pull/1700) (dependabot[bot])
- chore(deps): bump pytest from 9.0.2 to 9.0.3 in /python [#1702](https://github.com/apache/datafusion-ballista/pull/1702) (dependabot[bot])
- minor: change log level for few statements [#1706](https://github.com/apache/datafusion-ballista/pull/1706) (milenkovicm)
- chore(deps): bump config from 0.15.22 to 0.15.23 [#1709](https://github.com/apache/datafusion-ballista/pull/1709) (dependabot[bot])
- Saturate scheduler job elapsed time [#1708](https://github.com/apache/datafusion-ballista/pull/1708) (MukundaKatta)
- [TUI] Executor's id is not a numeric column. It should be center aligned [#1713](https://github.com/apache/datafusion-ballista/pull/1713) (martin-g)
- Make use of Swatinem/rust-cache to make the CI workflows faster [#1705](https://github.com/apache/datafusion-ballista/pull/1705) (martin-g)
- Merge Executor's brief and extended properties [#1716](https://github.com/apache/datafusion-ballista/pull/1716) (martin-g)
- [INFRA] Set up default rulesets for default and release branches [#1715](https://github.com/apache/datafusion-ballista/pull/1715) (asf-gitbox-commits)
- chore(deps): bump github/codeql-action from 4.35.4 to 4.35.5 [#1719](https://github.com/apache/datafusion-ballista/pull/1719) (dependabot[bot])
- chore(deps): bump ctor from 1.0.5 to 1.0.6 [#1720](https://github.com/apache/datafusion-ballista/pull/1720) (dependabot[bot])
- chore(deps): bump dashmap from 6.1.0 to 6.2.1 [#1721](https://github.com/apache/datafusion-ballista/pull/1721) (dependabot[bot])
- ci: add TPC-H SF10 workflow [#1688](https://github.com/apache/datafusion-ballista/pull/1688) (andygrove)
- minor: [TUI] Extract a helper method for the table/scrollbar area splitter [#1730](https://github.com/apache/datafusion-ballista/pull/1730) (martin-g)
- minor: [TUI] Sort the metrics before rendering them [#1731](https://github.com/apache/datafusion-ballista/pull/1731) (martin-g)
- chore: Do not run -cli tests twice [#1732](https://github.com/apache/datafusion-ballista/pull/1732) (martin-g)
- chore: generate changelog for ballista 53 [#1735](https://github.com/apache/datafusion-ballista/pull/1735) (milenkovicm)

## Credits

Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.

```
    74	dependabot[bot]
    29	Marko Milenković
    25	Andy Grove
    23	Martin Grigorov
     5	alexander domenti
     4	Kevin Liu
     3	Daniel Tu
     3	Saj
     1	Abhinav Gautam
     1	Harrison Crosse
     1	Mukunda Rao Katta
     1	The Apache Software Foundation
     1	gittihub-jpg
     1	goingforstudying-ctrl
     1	jgrim
     1	mete
```

Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.