Apache DataFusion Ballista 0.10.0 Changelog

Full Changelog

Implemented enhancements:

• Add user guide section on prometheus metrics #507

• Don’t throw error when job path not exist in remove_job_data #502

• Fix clippy warning #494

• Use job_data_clean_up_interval_seconds == 0 to indicate executor_cleanup_enable #488

• Add a config for tracing log rolling policy for both scheduler and executor #486

• Set up repo where we can push benchmark results #473

• Make the delayed time interval for cleanup job data in both scheduler and executor configurable #469

• Add some validation for the remove_job_data grpc service #467

• Add ability to build docker images using release-lto profile #463

• Suggest users download (rather than build) the FlightSQL JDBC Driver #460

• Clean up legacy job shuffle data #459

• Add grpc service for the scheduler to make it able to be triggered by client explicitly #458

• Replace Mutex<HashMap> by using DashMap #448

• Refine log level #446

• Upgrade to DataFusion 14.0.0 #445

• Add a feature for hdfs3 #419

• Add optional flag which advertises host for Arrow Flight SQL #418

• Partitioning reasoning in DataFusion and Ballista #284

• Stop wasting time in CI on MIRI runs #283

• Publish Docker images as part of each release #236

• Cleanup job/stage status from TaskManager and clean up shuffle data after a period after JobFinished #185

Fixed bugs:

• build broken: configure_me_codegen retroactively reserved bind_host #519

• Return empty results for SQLs with order by #451

• ballista scheduler is not taken inline parameters into account #443

• [FlightSQL] Cannot connect with Tableau Desktop #428

• Benchmark q15 fails #372

• Incorrect documentation for building Ballista on Linux when using docker-compose #362

• Scheduler silently replaces ParquetExec with EmptyExec if data path is not correctly mounted in container #353

• SQL with order by limit returns nothing #334

Documentation updates:

• README updates #433 (andygrove)

Merged pull requests:

• configure_me_codegen retroactively reserved on our bind_host parame… #520 (avantgardnerio)

• Bump actions/cache from 2 to 3 #517 (dependabot[bot])

• Update graphviz-rust requirement from 0.3.0 to 0.4.0 #515 (dependabot[bot])

• Add Prometheus metrics endpoint #511 (thinkharderdev)

• Enable tests that work since upgrading to DataFusion 14 #510 (andygrove)

• Update hashbrown requirement from 0.12 to 0.13 #506 (dependabot[bot])

• Don’t throw error when job shuffle data path not exist in executor #503 (yahoNanJing)

• Upgrade to DataFusion 14.0.0 and Arrow 26.0.0 #499 (andygrove)

• Fix clippy warning #495 (yahoNanJing)

• Stop wasting time in CI on MIRI runs #491 (Ted-Jiang)

• Remove executor config executor_cleanup_enable and make the configuation name for executor cleanup more intuitive #489 (yahoNanJing)

• Add a config for tracing log rolling policy for both scheduler and executor #487 (yahoNanJing)

• Add grpc service of cleaning up job shuffle data for the scheduler to make it able to be triggered by client explicitly #485 (yahoNanJing)

• [Minor] Bump DataFusion #480 (Dandandan)

• Remove benchmark results from README #478 (andygrove)

• Update flightsql.md to provide correct instruction #476 (iajoiner)

• Add support for Tableau #475 (avantgardnerio)

• Add SchedulerConfig for the scheduler configurations, like event_loop_buffer_size, finished_job_data_clean_up_interval_seconds, finished_job_state_clean_up_interval_seconds #472 (yahoNanJing)

• Bump DataFusion #471 (Dandandan)

• Add some validation for remove_job_data in the executor server #468 (yahoNanJing)

• Update documentation to reflect the release of the FlightSQL JDBC Driver #461 (avantgardnerio)

• Bump DataFusion version #453 (andygrove)

• Add shuffle for SortPreservingMergeExec physical operator #452 (yahoNanJing)

• Replace Mutex<HashMap> by using DashMap #449 (yahoNanJing)

• Refine log level for trial info and periodically invoked places #447 (yahoNanJing)

• MINOR: Add set -e to scripts, fix a typo #444 (andygrove)

• Add optional flag which advertises host for Arrow Flight SQL #418 #442 (DaltonModlin)

• Reorder joins after resolving stage inputs #441 (Dandandan)

• Add a feature for hdfs3 #439 (yahoNanJing)

• Add Spark benchmarks #438 (andygrove)

• scheduler now verifies that file:// ListingTable URLs are accessible #414 (andygrove)