Apache DataFusion Ballista 43.0.0 Changelog#

Full Changelog

Implemented enhancements:

  • feat: make max message size configurable for gRPC clients #983 (etolbakov)

  • feat: Upgrade to DataFusion 38 #1048 (andygrove)

  • feat: Upgrade to DataFusion 39.0.0 #1052 (andygrove)

  • feat: default instance for executor configuration #1147 (milenkovicm)

  • feat: Expose Ballista Scheduler and Executor in Python #1148 (milenkovicm)

  • feat: add test to check for ctx.enable_url_table() #1155 (milenkovicm)

Documentation updates:

  • docs: Add ASF attribution #973 (simicd)

  • Architecture guide #977 (andygrove)

  • docs: enhance the ballista-cli docs #979 (haoxins)

  • docs: update user guide with docker image information #980 (etolbakov)

  • docs: enhance the docs of Ballista client #985 (haoxins)

  • docs: list protoc dependency #989 (davidwilemski)

  • update asf yaml #1007 (andygrove)

  • docs: Add workflow to publish documentation #1040 (andygrove)

  • docs: Replace Arrow Ballista with DataFusion Ballista #1041 (andygrove)

  • Add maintenance status note #1043 (andygrove)

  • Remove helm from supported code #1071 (milenkovicm)

  • Remove UI #1072 (milenkovicm)

  • Remove HDFS support … #1073 (milenkovicm)

  • Removed Maintenance Notice #1094 (tbar4)

  • Update root README.md and other documentation with latest changes #1113 (milenkovicm)

  • docs: Update benchmarks #1121 (andygrove)

Merged pull requests:

  • PyBallista - Python SQL client for Ballista #970 (andygrove)

  • docs: Add ASF attribution #973 (simicd)

  • [Python] Add read_csv and read_parquet methods #976 (andygrove)

  • Architecture guide #977 (andygrove)

  • [Python] Add more methods to SessionContext #978 (andygrove)

  • [Python] Add execute_logical_plan to context #972 (andygrove)

  • Use correct product name in docs #975 (andygrove)

  • docs: enhance the ballista-cli docs #979 (haoxins)

  • docs: update user guide with docker image information #980 (etolbakov)

  • Upgrade Rust version to 1.72 to keep the same as DataFusion v35 #982 (haoxins)

  • build: Fix the ballista-cli Dockerfile #981 (haoxins)

  • feat: make max message size configurable for gRPC clients #983 (etolbakov)

  • Remove some hard-coded gRPC max message sizes #984 (andygrove)

  • docs: enhance the docs of Ballista client #985 (haoxins)

  • docs: list protoc dependency #989 (davidwilemski)

  • Fix ExecutorLost event debug info #988 (lewiszlw)

  • Fix shuffle writer test #998 (Jefffrey)

  • Bump graphviz-rust from 0.6.1 to 0.8.0 #999 (Jefffrey)

  • Add rust-toolchain.toml for clarity #1014 (scnerd)

  • Fix executor metadata decode bug #1004 (lewiszlw)

  • update asf yaml #1007 (andygrove)

  • Fix Ballista rust.yml github workflow #1026 (RaphaelMarinier)

  • Bump datafusion to 36.0.0 and make ballista compatible with it. #1027 (RaphaelMarinier)

  • Make Ballista compatible with Datafusion 37.0.0 (from 36.0.0) #1031 (RaphaelMarinier)

  • Fixes Setting Job Name Not Reflected in Ballista UI #1039 (athultr1997)

  • docs: Add workflow to publish documentation #1040 (andygrove)

  • [Docs] fix good_first_issue link in the contribution md doc #1022 (Almaz-KG)

  • docs: Replace Arrow Ballista with DataFusion Ballista #1041 (andygrove)

  • Fix job hangs when partition count of plan is zero #1024 (lewiszlw)

  • Add maintenance status note #1043 (andygrove)

  • Fix cargo build #1045 (andygrove)

  • fix docker build in CI #1046 (andygrove)

  • feat: Upgrade to DataFusion 38 #1048 (andygrove)

  • Bump actions/setup-node from 3 to 4 #909 (dependabot[bot])

  • Bump actions/cache from 3 to 4 #958 (dependabot[bot])

  • feat: Upgrade to DataFusion 39.0.0 #1052 (andygrove)

  • Update datafusion protobuf definitions #1057 (palaska)

  • Fix regression with TPC-H benchmark #1060 (andygrove)

  • Upgrade to Datafusion 41 #1062 (palaska)

  • Remove helm from supported code #1071 (milenkovicm)

  • Remove plugin subsystem #1070 (milenkovicm)

  • Remove CI folder #1074 (milenkovicm)

  • Code cleanup, move examples, remove unused files #1075 (milenkovicm)

  • Remove UI #1072 (milenkovicm)

  • Remove key-value stores for scheduler persistence #1077 (milenkovicm)

  • Remove cache functionality #1076 (milenkovicm)

  • Remove HDFS support … #1073 (milenkovicm)

  • Reorganise and remove dependencies #1078 (milenkovicm)

  • Promote keda and flight-sql to optional features #1079 (milenkovicm)

  • Update to datafusion 42 … #1080 (milenkovicm)

  • #1086 solve examples errors #1087 (tbar4)

  • fix issue with not building python package … #1085 (milenkovicm)

  • another round of code cleanup … #1089 (milenkovicm)

  • Make rest-api optional feature … #1084 (milenkovicm)

  • fix clippy issues after updating to rust 1.82 #1090 (milenkovicm)

  • Replace BallistaContext with SessionContext #1088 (milenkovicm)

  • Removed Maintenance Notice #1094 (tbar4)

  • Ergonomic way to setup/configure SessionContextExt #1096 (milenkovicm)

  • Executor configuration extended .. #1099 (milenkovicm)

  • fix issue with executor registration … #1101 (milenkovicm)

  • Deprecate BallistaContext #1103 (milenkovicm)

  • fix imports after un-rebased PR #1106 (milenkovicm)

  • Ballista proto cleanup #1110 (milenkovicm)

  • Update and move deps to workspace #1109 (milenkovicm)

  • Trim down BallistaConfig #1108 (milenkovicm)

  • Remove build-in object store registry #1114 (milenkovicm)

  • Update root README.md and other documentation with latest changes #1113 (milenkovicm)

  • support window functions #1112 (onursatici)

  • added a BallistaContext to ballista to allow for Remote or standalone #1100 (tbar4)

  • Decommission BallistaContext #1119 (milenkovicm)

  • docs: Update benchmarks #1121 (andygrove)

  • Make easier to create custom schedulers and executors #1118 (milenkovicm)

  • refactor: Move BallistaRegistry to better location #1126 (milenkovicm)

  • refactor: BallistaLogicalExtensionCodec refactoring and improvements #1127 (milenkovicm)

  • refactor: consolidate ballista tests #1129 (milenkovicm)

  • refactor: SessionStateExt and SessionConfigExt #1130 (milenkovicm)

  • chore: dependancy updates #1131 (milenkovicm)

  • chore: fix warning mimaloc warning when building #1137 (milenkovicm)

  • refactor: SessionBuilder to return Result<_> #1138 (milenkovicm)

  • chore: remove unused cache_ options from executor #1140 (milenkovicm)

  • updated maturin version and ccargo build to build yml #1136 (tbar4)

  • chore: Fix clippy issues after rust update (1.83.0) #1143 (milenkovicm)

  • Fix documentation example which still uses BallistaContext #1145 (milenkovicm)

  • Ballista proto cleanup #1146 (milenkovicm)

  • feat: default instance for executor configuration #1147 (milenkovicm)

  • feat: Expose Ballista Scheduler and Executor in Python #1148 (milenkovicm)

  • chore: dependency cleanup #1150 (milenkovicm)

  • Update DataFusion to 43 #1125 (Dandandan)

  • Reinstantiate join order optimization #1122 (Dandandan)

  • add partitioning scheme for unresolved shuffle and shuffle reader exec #1144 (onursatici)

  • chore: update py-df to 43.1 #1152 (milenkovicm)

  • chore: no need to run python test in rust #1154 (milenkovicm)

  • feat: add test to check for ctx.enable_url_table() #1155 (milenkovicm)