Testing#

Tests are critical to ensure that DataFusion is working properly and is not accidentally broken during refactorings. All new features should have test coverage and the entire test suite is run as part of CI.

For CI workflows, documentation checks, and benchmarks, see Testing Infrastructure.

Testing Quick Start#

While developing a feature or bug fix, best practice is to run the smallest set of tests that gives confidence for your change, then expand as needed.

Initially, run the tests in the crates you changed. For example, if you made changes to files in datafusion-optimizer/src, run the corresponding crate tests:

cargo test -p datafusion-optimizer

Then, run the sqllogictest suite, which provides a strong speed–coverage tradeoff for development: it runs quickly while offering broad regression coverage across most SQL behavior in DataFusion.

cargo test --profile=ci --test sqllogictests

Finally, before submitting a PR, run the tests for the core datafusion and datafusion-cli crates:

cargo test -p datafusion
cargo test -p datafusion-cli

Some integration tests require optional external services such as Docker-backed containers and may skip when unavailable.

Testing Overview#

DataFusion has several levels of tests in its Test Pyramid and tries to follow the Rust standard Testing Organization described in The Book.

Run tests using cargo:

cargo test

You can also use other runners such as cargo-nextest.

cargo nextest run

Code coverage#

The Codecov bot posts a coverage report on GitHub pull requests with:

  • all-tests: coverage from the workspace test suite.

  • sqllogictests: coverage from SQLLogicTests alone.

We prefer end-to-end tests because they are easier to maintain and verify final results through the full query pipeline, including optimizer passes. The SLT report helps ensure good coverage from these tests alone.

Coverage helps find missing tests; high coverage does not guarantee good tests:

  • Do: Enumerate edge cases to build comprehensive tests, then check the coverage report for gaps.

  • Don’t: Use Codecov to choose a minimal set of tests that covers a new feature.

Choosing What Kind of Test to Write#

When adding tests, prefer tests that exercise user-visible behavior over tests of internal implementation details. Doing so means your tests are more likely to keep working (and keep catching regressions) while the code is refactored. In rough order of preference:

  1. Use sqllogictest (.slt) tests for any behavior that can be expressed in SQL.

  2. End-to-end tests that use public APIs such as the DataFrame API, for behavior that cannot be expressed in SQL. For example, DataFrame tests live in datafusion/core/tests/dataframe and run as part of the core_integration test binary:

    cargo test -p datafusion --test core_integration -- dataframe

  3. Rust unit tests for logic that cannot practically be reached through SQL or DataFrame public APIs.

Unit tests#

Tests for code in an individual module are defined in the same source file with a test module, following Rust convention.

For example, to run tests in the datafusion crate:

cargo test -p datafusion

The test_util module provides useful macros to write unit tests effectively, such as assert_batches_sorted_eq and assert_batches_eq for RecordBatches and assert_contains / assert_not_contains which are used extensively in the codebase.

sqllogictests Tests#

DataFusion’s SQL implementation is tested using sqllogictest. You can run these tests with commands like:

# Run all tests
cargo test --profile=ci --test sqllogictests
# Run a specific test file
cargo test --profile=ci --test sqllogictests -- aggregate.slt
# Run a specific test file and update expected outputs
cargo test --profile=ci --test sqllogictests -- aggregate.slt --complete
# Run and update expected outputs for all test files
cargo test --profile=ci --test sqllogictests -- --complete

sqllogictests may be less convenient for new contributors who are familiar with writing .rs tests as they require learning another tool. However, sqllogictest based tests are much easier to develop and maintain as they 1) do not require a slow recompile/link cycle and 2) can be automatically updated.

Like similar systems such as DuckDB, DataFusion has chosen to trade off a slightly higher barrier to contribution for longer term maintainability.

DataFusion runs sqlite’s test suite in the merge queue before merging PRs into main. For local instructions, see Running Tests: sqlite.

Snapshot testing (cargo insta)#

Insta is used for snapshot testing. Snapshots are generated and compared on each test run. If the output changes, tests will fail.

To review the changes, you can use Insta CLI:

cargo install cargo-insta
cargo insta review

Extended Tests#

DataFusion has extended tests (defined in extended.yml) that take significantly longer to run than the standard suite. They provide additional correctness coverage and must pass in the merge queue before a PR merges into main.

To conserve CI resources, these tests do not run on ordinary PR updates. They also run on pushes to release branches (branch-*). You can run them manually.

For local SQLite test instructions, see the instructions in the documentation.

Rust Integration Tests#

There are several public interface tests for the DataFusion library in the tests directory.

You can run these tests individually using cargo as normal command such as

cargo test -p datafusion --test parquet_integration

Fuzz testing#

Fuzz testing generates random inputs and configurations, then checks that a property holds for every generated case. For example, an optimized operator should produce the same results as a reference implementation.

Fuzzing unit tests#

For fuzzing unit tests, such as tests for a single operator or an internal utility:

  • If the test is small (easy to set up and takes a few seconds), keep it as a unit test.

  • If the test is large (requires a fuzzing driver and takes more than 10 seconds), it should implement the FuzzTest trait and run through the central fuzz-test entry point. See the spill-pool fuzzer for a reference implementation.

SQL fuzz testing#

For SQL fuzzing of more complex queries, use datafusion-fuzzer, which is currently maintained outside the core DataFusion repository. See its documentation for details.