<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->

# Apache DataFusion Ballista 43.0.0 Changelog

[Full Changelog](https://github.com/apache/datafusion-ballista/compare/0.12.0...43.0.0-rc2)

**Implemented enhancements:**

- feat: make max message size configurable for gRPC clients [#983](https://github.com/apache/datafusion-ballista/pull/983) (etolbakov)
- feat: Upgrade to DataFusion 38 [#1048](https://github.com/apache/datafusion-ballista/pull/1048) (andygrove)
- feat: Upgrade to DataFusion 39.0.0 [#1052](https://github.com/apache/datafusion-ballista/pull/1052) (andygrove)
- feat: default instance for executor configuration [#1147](https://github.com/apache/datafusion-ballista/pull/1147) (milenkovicm)
- feat: Expose Ballista Scheduler and Executor in Python [#1148](https://github.com/apache/datafusion-ballista/pull/1148) (milenkovicm)
- feat: add test to check for `ctx.enable_url_table()` [#1155](https://github.com/apache/datafusion-ballista/pull/1155) (milenkovicm)

**Documentation updates:**

- docs: Add ASF attribution [#973](https://github.com/apache/datafusion-ballista/pull/973) (simicd)
- Architecture guide [#977](https://github.com/apache/datafusion-ballista/pull/977) (andygrove)
- docs: enhance the ballista-cli docs [#979](https://github.com/apache/datafusion-ballista/pull/979) (haoxins)
- docs: update user guide with docker image information [#980](https://github.com/apache/datafusion-ballista/pull/980) (etolbakov)
- docs: enhance the docs of Ballista client [#985](https://github.com/apache/datafusion-ballista/pull/985) (haoxins)
- docs: list protoc dependency [#989](https://github.com/apache/datafusion-ballista/pull/989) (davidwilemski)
- update asf yaml [#1007](https://github.com/apache/datafusion-ballista/pull/1007) (andygrove)
- docs: Add workflow to publish documentation [#1040](https://github.com/apache/datafusion-ballista/pull/1040) (andygrove)
- docs: Replace Arrow Ballista with DataFusion Ballista [#1041](https://github.com/apache/datafusion-ballista/pull/1041) (andygrove)
- Add maintenance status note [#1043](https://github.com/apache/datafusion-ballista/pull/1043) (andygrove)
- Remove helm from supported code [#1071](https://github.com/apache/datafusion-ballista/pull/1071) (milenkovicm)
- Remove UI [#1072](https://github.com/apache/datafusion-ballista/pull/1072) (milenkovicm)
- Remove HDFS support ... [#1073](https://github.com/apache/datafusion-ballista/pull/1073) (milenkovicm)
- Removed Maintenance Notice [#1094](https://github.com/apache/datafusion-ballista/pull/1094) (tbar4)
- Update root `README.md` and other documentation with latest changes [#1113](https://github.com/apache/datafusion-ballista/pull/1113) (milenkovicm)
- docs: Update benchmarks [#1121](https://github.com/apache/datafusion-ballista/pull/1121) (andygrove)

**Merged pull requests:**

- PyBallista - Python SQL client for Ballista [#970](https://github.com/apache/datafusion-ballista/pull/970) (andygrove)
- docs: Add ASF attribution [#973](https://github.com/apache/datafusion-ballista/pull/973) (simicd)
- [Python] Add `read_csv` and `read_parquet` methods [#976](https://github.com/apache/datafusion-ballista/pull/976) (andygrove)
- Architecture guide [#977](https://github.com/apache/datafusion-ballista/pull/977) (andygrove)
- [Python] Add more methods to SessionContext [#978](https://github.com/apache/datafusion-ballista/pull/978) (andygrove)
- [Python] Add `execute_logical_plan` to context [#972](https://github.com/apache/datafusion-ballista/pull/972) (andygrove)
- Use correct product name in docs [#975](https://github.com/apache/datafusion-ballista/pull/975) (andygrove)
- docs: enhance the ballista-cli docs [#979](https://github.com/apache/datafusion-ballista/pull/979) (haoxins)
- docs: update user guide with docker image information [#980](https://github.com/apache/datafusion-ballista/pull/980) (etolbakov)
- Upgrade Rust version to 1.72 to keep the same as DataFusion v35 [#982](https://github.com/apache/datafusion-ballista/pull/982) (haoxins)
- build: Fix the ballista-cli Dockerfile [#981](https://github.com/apache/datafusion-ballista/pull/981) (haoxins)
- feat: make max message size configurable for gRPC clients [#983](https://github.com/apache/datafusion-ballista/pull/983) (etolbakov)
- Remove some hard-coded gRPC max message sizes [#984](https://github.com/apache/datafusion-ballista/pull/984) (andygrove)
- docs: enhance the docs of Ballista client [#985](https://github.com/apache/datafusion-ballista/pull/985) (haoxins)
- docs: list protoc dependency [#989](https://github.com/apache/datafusion-ballista/pull/989) (davidwilemski)
- Fix ExecutorLost event debug info [#988](https://github.com/apache/datafusion-ballista/pull/988) (lewiszlw)
- Fix shuffle writer test [#998](https://github.com/apache/datafusion-ballista/pull/998) (Jefffrey)
- Bump graphviz-rust from 0.6.1 to 0.8.0 [#999](https://github.com/apache/datafusion-ballista/pull/999) (Jefffrey)
- Add rust-toolchain.toml for clarity [#1014](https://github.com/apache/datafusion-ballista/pull/1014) (scnerd)
- Fix executor metadata decode bug [#1004](https://github.com/apache/datafusion-ballista/pull/1004) (lewiszlw)
- update asf yaml [#1007](https://github.com/apache/datafusion-ballista/pull/1007) (andygrove)
- Fix Ballista rust.yml github workflow [#1026](https://github.com/apache/datafusion-ballista/pull/1026) (RaphaelMarinier)
- Bump datafusion to 36.0.0 and make ballista compatible with it. [#1027](https://github.com/apache/datafusion-ballista/pull/1027) (RaphaelMarinier)
- Make Ballista compatible with Datafusion 37.0.0 (from 36.0.0) [#1031](https://github.com/apache/datafusion-ballista/pull/1031) (RaphaelMarinier)
- Fixes Setting Job Name Not Reflected in Ballista UI [#1039](https://github.com/apache/datafusion-ballista/pull/1039) (athultr1997)
- docs: Add workflow to publish documentation [#1040](https://github.com/apache/datafusion-ballista/pull/1040) (andygrove)
- [Docs] fix good_first_issue link in the contribution md doc [#1022](https://github.com/apache/datafusion-ballista/pull/1022) (Almaz-KG)
- docs: Replace Arrow Ballista with DataFusion Ballista [#1041](https://github.com/apache/datafusion-ballista/pull/1041) (andygrove)
- Fix job hangs when partition count of plan is zero [#1024](https://github.com/apache/datafusion-ballista/pull/1024) (lewiszlw)
- Add maintenance status note [#1043](https://github.com/apache/datafusion-ballista/pull/1043) (andygrove)
- Fix cargo build [#1045](https://github.com/apache/datafusion-ballista/pull/1045) (andygrove)
- fix docker build in CI [#1046](https://github.com/apache/datafusion-ballista/pull/1046) (andygrove)
- feat: Upgrade to DataFusion 38 [#1048](https://github.com/apache/datafusion-ballista/pull/1048) (andygrove)
- Bump actions/setup-node from 3 to 4 [#909](https://github.com/apache/datafusion-ballista/pull/909) (dependabot[bot])
- Bump actions/cache from 3 to 4 [#958](https://github.com/apache/datafusion-ballista/pull/958) (dependabot[bot])
- feat: Upgrade to DataFusion 39.0.0 [#1052](https://github.com/apache/datafusion-ballista/pull/1052) (andygrove)
- Update datafusion protobuf definitions [#1057](https://github.com/apache/datafusion-ballista/pull/1057) (palaska)
- Fix regression with TPC-H benchmark [#1060](https://github.com/apache/datafusion-ballista/pull/1060) (andygrove)
- Upgrade to Datafusion 41 [#1062](https://github.com/apache/datafusion-ballista/pull/1062) (palaska)
- Remove helm from supported code [#1071](https://github.com/apache/datafusion-ballista/pull/1071) (milenkovicm)
- Remove plugin subsystem [#1070](https://github.com/apache/datafusion-ballista/pull/1070) (milenkovicm)
- Remove CI folder [#1074](https://github.com/apache/datafusion-ballista/pull/1074) (milenkovicm)
- Code cleanup, move examples, remove unused files [#1075](https://github.com/apache/datafusion-ballista/pull/1075) (milenkovicm)
- Remove UI [#1072](https://github.com/apache/datafusion-ballista/pull/1072) (milenkovicm)
- Remove key-value stores for scheduler persistence [#1077](https://github.com/apache/datafusion-ballista/pull/1077) (milenkovicm)
- Remove cache functionality [#1076](https://github.com/apache/datafusion-ballista/pull/1076) (milenkovicm)
- Remove HDFS support ... [#1073](https://github.com/apache/datafusion-ballista/pull/1073) (milenkovicm)
- Reorganise and remove dependencies [#1078](https://github.com/apache/datafusion-ballista/pull/1078) (milenkovicm)
- Promote keda and flight-sql to optional features [#1079](https://github.com/apache/datafusion-ballista/pull/1079) (milenkovicm)
- Update to datafusion 42 ... [#1080](https://github.com/apache/datafusion-ballista/pull/1080) (milenkovicm)
- #1086 solve examples errors [#1087](https://github.com/apache/datafusion-ballista/pull/1087) (tbar4)
- fix issue with not building python package ... [#1085](https://github.com/apache/datafusion-ballista/pull/1085) (milenkovicm)
- another round of code cleanup ... [#1089](https://github.com/apache/datafusion-ballista/pull/1089) (milenkovicm)
- Make rest-api optional feature ... [#1084](https://github.com/apache/datafusion-ballista/pull/1084) (milenkovicm)
- fix clippy issues after updating to rust 1.82 [#1090](https://github.com/apache/datafusion-ballista/pull/1090) (milenkovicm)
- Replace BallistaContext with SessionContext [#1088](https://github.com/apache/datafusion-ballista/pull/1088) (milenkovicm)
- Removed Maintenance Notice [#1094](https://github.com/apache/datafusion-ballista/pull/1094) (tbar4)
- Ergonomic way to setup/configure `SessionContextExt` [#1096](https://github.com/apache/datafusion-ballista/pull/1096) (milenkovicm)
- Executor configuration extended .. [#1099](https://github.com/apache/datafusion-ballista/pull/1099) (milenkovicm)
- fix issue with executor registration ... [#1101](https://github.com/apache/datafusion-ballista/pull/1101) (milenkovicm)
- Deprecate `BallistaContext` [#1103](https://github.com/apache/datafusion-ballista/pull/1103) (milenkovicm)
- fix imports after un-rebased PR [#1106](https://github.com/apache/datafusion-ballista/pull/1106) (milenkovicm)
- Ballista proto cleanup [#1110](https://github.com/apache/datafusion-ballista/pull/1110) (milenkovicm)
- Update and move deps to workspace [#1109](https://github.com/apache/datafusion-ballista/pull/1109) (milenkovicm)
- Trim down `BallistaConfig` [#1108](https://github.com/apache/datafusion-ballista/pull/1108) (milenkovicm)
- Remove build-in object store registry [#1114](https://github.com/apache/datafusion-ballista/pull/1114) (milenkovicm)
- Update root `README.md` and other documentation with latest changes [#1113](https://github.com/apache/datafusion-ballista/pull/1113) (milenkovicm)
- support window functions [#1112](https://github.com/apache/datafusion-ballista/pull/1112) (onursatici)
- added a BallistaContext to ballista to allow for Remote or standalone [#1100](https://github.com/apache/datafusion-ballista/pull/1100) (tbar4)
- Decommission `BallistaContext` [#1119](https://github.com/apache/datafusion-ballista/pull/1119) (milenkovicm)
- docs: Update benchmarks [#1121](https://github.com/apache/datafusion-ballista/pull/1121) (andygrove)
- Make easier to create custom schedulers and executors [#1118](https://github.com/apache/datafusion-ballista/pull/1118) (milenkovicm)
- refactor: Move BallistaRegistry to better location [#1126](https://github.com/apache/datafusion-ballista/pull/1126) (milenkovicm)
- refactor: BallistaLogicalExtensionCodec refactoring and improvements [#1127](https://github.com/apache/datafusion-ballista/pull/1127) (milenkovicm)
- refactor: consolidate ballista tests [#1129](https://github.com/apache/datafusion-ballista/pull/1129) (milenkovicm)
- refactor: SessionStateExt and SessionConfigExt [#1130](https://github.com/apache/datafusion-ballista/pull/1130) (milenkovicm)
- chore: dependancy updates [#1131](https://github.com/apache/datafusion-ballista/pull/1131) (milenkovicm)
- chore: fix warning mimaloc warning when building [#1137](https://github.com/apache/datafusion-ballista/pull/1137) (milenkovicm)
- refactor: SessionBuilder to return Result<\_> [#1138](https://github.com/apache/datafusion-ballista/pull/1138) (milenkovicm)
- chore: remove unused cache\_ options from executor [#1140](https://github.com/apache/datafusion-ballista/pull/1140) (milenkovicm)
- updated maturin version and ccargo build to build yml [#1136](https://github.com/apache/datafusion-ballista/pull/1136) (tbar4)
- chore: Fix clippy issues after rust update (1.83.0) [#1143](https://github.com/apache/datafusion-ballista/pull/1143) (milenkovicm)
- Fix documentation example which still uses BallistaContext [#1145](https://github.com/apache/datafusion-ballista/pull/1145) (milenkovicm)
- Ballista proto cleanup [#1146](https://github.com/apache/datafusion-ballista/pull/1146) (milenkovicm)
- feat: default instance for executor configuration [#1147](https://github.com/apache/datafusion-ballista/pull/1147) (milenkovicm)
- feat: Expose Ballista Scheduler and Executor in Python [#1148](https://github.com/apache/datafusion-ballista/pull/1148) (milenkovicm)
- chore: dependency cleanup [#1150](https://github.com/apache/datafusion-ballista/pull/1150) (milenkovicm)
- Update DataFusion to 43 [#1125](https://github.com/apache/datafusion-ballista/pull/1125) (Dandandan)
- Reinstantiate join order optimization [#1122](https://github.com/apache/datafusion-ballista/pull/1122) (Dandandan)
- add partitioning scheme for unresolved shuffle and shuffle reader exec [#1144](https://github.com/apache/datafusion-ballista/pull/1144) (onursatici)
- chore: update py-df to 43.1 [#1152](https://github.com/apache/datafusion-ballista/pull/1152) (milenkovicm)
- chore: no need to run python test in rust [#1154](https://github.com/apache/datafusion-ballista/pull/1154) (milenkovicm)
- feat: add test to check for `ctx.enable_url_table()` [#1155](https://github.com/apache/datafusion-ballista/pull/1155) (milenkovicm)