Apache DataFusion Ballista 0.11.0 Changelog#

Full Changelog

Implemented enhancements:

  • Remove python since it has been moved to its own repo, datafusion-ballista-python #653

  • Add executor self-registration mechanism in the heartbeat service #648

  • Upgrade to DataFusion 17 #638

  • Move Python bindings to separate repo? #635

  • Implement new release process #622

  • Change default branch name from master to main #618

  • Update latest datafusion dependency #610

  • Implement optimizer rule to remove redundant repartitioning #608

  • ballista-cli as (docker) images #600

  • Update contributor guide #598

  • Fix cargo clippy #570

  • Support Alibaba Cloud OSS with ObjectStore #566

  • Refactor StateBackendClient to be a higher-level interface #554

  • Make it concurrently to launch tasks to executors #544

  • Simplify docs #531

  • Provide an in-memory StateBackend #505

  • Add support for Azure blob storage #294

  • Add a workflow to build the image and publish it to the package #71

Fixed bugs:

  • Rust / Check Cargo.toml formatting (amd64, stable) (pull_request) Failing #662

  • Protobuf parsing error #646

  • jobs from python client not showing up in Scheduler UI #625

  • ballista ui fails to build #594

  • cargo build –release fails for ballista-scheduler #590

  • docker build fails #589

  • Multi-scheduler Job Starvation #585

  • Cannot query file from S3 #559

  • Benchmark q16 fails #373

Documentation updates:

Merged pull requests: