Introduction¶
DataFusion is a very fast, extensible query engine for building high-quality data-centric systems in Rust, using the Apache Arrow in-memory format. DataFusion originated as part of the Apache Arrow project.
DataFusion offers SQL and Dataframe APIs, excellent performance, built-in support for CSV, Parquet, JSON, and Avro, python bindings, extensive customization, a great community, and more.
Project Goals¶
DataFusion aims to be the query engine of choice for new, fast data centric systems such as databases, dataframe libraries, machine learning and streaming applications by leveraging the unique features of Rust and Apache Arrow.
Features¶
Feature-rich SQL support and DataFrame API
Blazingly fast, vectorized, multi-threaded, streaming execution engine.
Native support for Parquet, CSV, JSON, and Avro file formats. Support for custom file formats and non file datasources via the
TableProvider
trait.Many extension points: user defined scalar/aggregate/window functions, DataSources, SQL, other query languages, custom plan and execution nodes, optimizer passes, and more.
Streaming, asynchronous IO directly from popular object stores, including AWS S3, Azure Blob Storage, and Google Cloud Storage (Other storage systems are supported via the
ObjectStore
trait).A state of the art query optimizer with expression coercion and simplification, projection and filter pushdown, sort and distribution aware optimizations, automatic join reordering, and more.
Permissive Apache 2.0 License, predictable and well understood Apache Software Foundation governance.
Implementation in Rust, a modern system language with development productivity similar to Java or Golang, the performance of C++, and loved by programmers everywhere.
Support for Substrait query plans, to easily pass plans across language and system boundaries.
Use Cases¶
DataFusion can be used without modification as an embedded SQL engine or can be customized and used as a foundation for building new systems.
While most current usecases are “analytic” or (throughput) some components of DataFusion such as the plan representations, are suitable for “streaming” and “transaction” style systems (low latency).
Here are some example systems built using DataFusion:
Specialized Analytical Database systems such as HoraeDB and more general Apache Spark like system such a Ballista.
New query language engines such as prql-query and accelerators such as VegaFusion
Research platform for new Database Systems, such as Flock
SQL support to another library, such as dask sql
Streaming data platforms such as Synnada
Tools for reading / sorting / transcoding Parquet, CSV, AVRO, and JSON files such as qv
Native Spark runtime replacement such as Blaze
By using DataFusion, projects are freed to focus on their specific features, and avoid reimplementing general (but still necessary) features such as an expression representation, standard optimizations, parellelized streaming execution plans, file format support, etc.
Known Users¶
Here are some active projects using DataFusion:
Arroyo Distributed stream processing engine in Rust
Ballista Distributed SQL Query Engine
Blaze The Blaze accelerator for Apache Spark leverages native vectorized execution to accelerate query processing
CnosDB Open Source Distributed Time Series Database
Comet Apache Spark native query execution plugin
Dask SQL Distributed SQL query engine in Python
delta-rs Native Rust implementation of Delta Lake
Exon Analysis toolkit for life-science applications
GlareDB Fast SQL database for querying and analyzing distributed data.
GreptimeDB Open Source & Cloud Native Distributed Time Series Database
HoraeDB Distributed Time-Series Database
InfluxDB Time Series Database
Kamu Planet-scale streaming data pipeline
LakeSoul Open source LakeHouse framework with native IO in Rust.
Lance Modern columnar data format for ML
OpenObserve Distributed cloud native observability platform
ParadeDB PostgreSQL for Search & Analytics
Parseable Log storage and observability platform
qv Quickly view your data
Restate Easily build resilient applications using distributed durable async/await
Sail Unifying stream, batch, and AI workloads with Apache Spark compatibility
Seafowl CDN-friendly analytical database
Spice.ai Unified SQL query interface & materialization engine
Synnada Streaming-first framework for data products
VegaFusion Server-side acceleration for the Vega visualization grammar
Here are some less active projects that used DataFusion:
bdt Boring Data Tool
datafusion-tui Text UI for DataFusion
Integrations and Extensions¶
There are a number of community projects that extend DataFusion or provide integrations with other systems, some of which are described below:
Language Bindings¶
Integrations¶
Why DataFusion?¶
High Performance: Leveraging Rust and Arrow’s memory model, DataFusion is very fast.
Easy to Connect: Being part of the Apache Arrow ecosystem (Arrow, Parquet and Flight), DataFusion works well with the rest of the big data ecosystem
Easy to Embed: Allowing extension at almost any point in its design, and published regularly as a crate on crates.io, DataFusion can be integrated and tailored for your specific usecase.
High Quality: Extensively tested, both by itself and with the rest of the Arrow ecosystem, DataFusion can and is used as the foundation for production systems.