Apache DataFusion Ballista 0.6.0 Changelog

Full Changelog

Breaking changes:

• File partitioning for ListingTable #1141 (rdettai)

• Register tables in BallistaContext using TableProviders instead of Dataframe #1028 (rdettai)

• Make TableProvider.scan() and PhysicalPlanner::create_physical_plan() async #1013 (rdettai)

• Reorganize table providers by table format #1010 (rdettai)

• Move CBOs and Statistics to physical plan #965 (rdettai)

• Update to sqlparser v 0.10.0 #934 [sql] (alamb)

• FilePartition and PartitionedFile for scanning flexibility #932 [sql] (yjshen)

• Improve SQLMetric APIs, port existing metrics #908 (alamb)

• Add support for EXPLAIN ANALYZE #858 [sql] (alamb)

• Rename concurrency to target_partitions #706 (andygrove)

Implemented enhancements:

• Update datafusion-cli to support Ballista, or implement new ballista-cli #886

• Prepare Ballista crates for publishing #509

• Add drop table support #1266 [sql] (viirya)

• use arrow 6.1.0 #1255 (Jimexist)

• Add support for create table as via MemTable #1243 [sql] (Dandandan)

• add values list expression #1165 [sql] (Jimexist)

• Multiple files per partitions for CSV Avro Json #1138 (rdettai)

• Implement INTERSECT & INTERSECT DISTINCT #1135 [sql] (xudong963)

• Simplify file struct abstractions #1120 (rdettai)

• Implement is [not] distinct from #1117 [sql] (Dandandan)

• add digest(utf8, method) function and refactor all current hash digest functions #1090 (Jimexist)

• [crypto] add blake3 algorithm to digest function #1086 (Jimexist)

• [crypto] add blake2b and blake2s functions #1081 (Jimexist)

• Update sqlparser-rs to 0.11 #1052 [sql] (alamb)

• remove hard coded partition count in ballista logicalplan deserialization #1044 (xudong963)

• Indexed field access for List #1006 [sql] (Igosuki)

• Update DataFusion to arrow 6.0 #984 (alamb)

• Implement Display for Expr, improve operator display #971 [sql] (matthewmturner)

• ObjectStore API to read from remote storage systems #950 (yjshen)

• fixes #933 replace placeholder fmt_as fr ExecutionPlan impls #939 (tiphaineruy)

• Support NotLike in Ballista #916 (Dandandan)

• Avro Table Provider #910 [sql] (Igosuki)

• Add BaselineMetrics, Timestamp metrics, add for CoalescePartitionsExec, rename output_time -> elapsed_compute #909 (alamb)

• [Ballista] Add executor last seen info to the ui #895 (msathis)

• add cross join support to ballista #891 (houqp)

• Add Ballista support to DataFusion CLI #889 (andygrove)

• Add support for PostgreSQL regex match #870 [sql] (b41sh)

Fixed bugs:

• Test execution_plans::shuffle_writer::tests::test Fail #1040

• Integration test fails to build docker images #918

• Ballista: Remove hard-coded concurrency from logical plan serde code #708

• How can I make ballista distributed compute work? #327

• fix subquery alias #1067 [sql] (xudong963)

• Fix compilation for ballista in stand-alone mode #1008 (Igosuki)

Documentation updates:

• Add Ballista roadmap #1166 (andygrove)

• Adds note on compatible rust version #1097 (1nF0rmed)

• implement approx_distinct function using HyperLogLog #1087 (Jimexist)

• Improve User Guide #954 (andygrove)

• Update plan_query_stages doc #951 (rdettai)

• [DataFusion] - Add show and show_limit function for DataFrame #923 (francis-du)

• update docs related to protoc and optional syntax #902 (Jimexist)

• Improve Ballista crate README content #878 (andygrove)

Performance improvements:

• optimize build profile for datafusion python binding, cli and ballista #1137 (houqp)

Closed issues:

• InList expr with NULL literals do not work #1190

• update the homepage README to include values, approx_distinct, etc. #1171

• [Python]: Inconsistencies with Python package name #1011

• Wanting to contribute to project where to start? #983

• delete redundant code #973

• How to build DataFusion python wheel #853

• Produce a design for a metrics framework #21

Merged pull requests:

• [nit] simplify ballista executor CollectExec impl codes #1140 (panarch)

For older versions, see apache/arrow/CHANGELOG.md