Testing Infrastructure#

This page covers checks for CI workflows, documentation, repository maintenance, and benchmarks. For tests of DataFusion’s query behavior, see Testing.

CI Workflow Tests#

The Rust workflow validator tests and fork status bridge tests run in CI and in ./dev/rust_lint.sh. Run them locally:

python3 ci/scripts/test_rust_workflow_skips.py
node --test ci/scripts/test_fork_ci_status.cjs

Documentation Examples#

We use Rust doctest to verify examples from the documentation are correct and up-to-date. These tests are run as part of our CI and you can run them locally with the following command:

cargo test --doc

API Documentation Examples#

As with other Rust projects, examples in doc comments in .rs files are automatically checked to ensure they work and evolve along with the code.

Prefix a line with # (a # followed by a space) to keep it compiled and run, but hidden from the rendered documentation. This is most useful for use statements, since readers care about the API being demonstrated, not its import boilerplate:

///
```
/// # use std::sync::Arc;
/// let value = Arc::new(42);
/// ```

Only the let value = ... line renders.

User Guide Documentation#

Rust example code from the user guide (anything marked with ```rust) is also tested in the same way using the doc_comment crate. See the end of core/src/lib.rs for more details.

Documentation Link Checks#

ci/scripts/markdown_link_check.sh checks internal Markdown links and runs as part of ./dev/rust_lint.sh. To run it on its own:

bash ci/scripts/markdown_link_check.sh

Rust doc comments are validated by rustdoc in CI and can be checked locally with:

bash ci/scripts/rust_docs.sh

ASF Status Check Validation#

ci/scripts/check_asf_yaml_status_checks.py checks that every required status check in .asf.yaml matches a job in .github/workflows. ci/scripts/check_rust_workflow_skips.py checks that rust.yml skips only its listed jobs on pushes to main and on pull requests from forks. ./dev/rust_lint.sh runs both checks. To run them on their own:

uv run python3 ci/scripts/check_asf_yaml_status_checks.py
uv run python3 ci/scripts/check_rust_workflow_skips.py

Security Audit#

ci/scripts/security_audit.sh runs cargo audit on the root Cargo.lock with the advisory exceptions that CI uses. ./dev/rust_lint.sh runs it and installs cargo-audit if it is missing. To run the audit on its own:

./ci/scripts/security_audit.sh

The audit fetches the RustSec advisory database. A new advisory or a different cargo-audit version can change the result without any change to the repository.

Large File Check#

ci/scripts/check_large_files.sh fails if any file committed between a base ref and a head ref is larger than 1.5 MB, the same check the “Large files PR check” workflow runs on pull requests. To run the check on its own, or against a different range:

./ci/scripts/check_large_files.sh
./ci/scripts/check_large_files.sh --base upstream/main --head my-branch

Only committed files are checked. Commit a change before running it.

Dependency Checks#

CI runs two dependency checks, and ./dev/rust_lint.sh runs both:

ci/scripts/check_circular_dependencies.sh fails on dependency cycles between DataFusion crates, including one through a dev-dependency, which Cargo allows when building but which prevents publishing the crates.
ci/scripts/check_unused_dependencies.sh runs cargo machete --with-metadata from the repository root.

To run either check on its own:

./ci/scripts/check_circular_dependencies.sh
./ci/scripts/check_unused_dependencies.sh

Examples README Check#

datafusion-examples/README.md is generated from the documentation comments in datafusion-examples/examples/<group>/main.rs. ci/scripts/check_examples_docs.sh regenerates it and fails if the committed file differs. To run the check on its own, or to update the README:

./ci/scripts/check_examples_docs.sh
./ci/scripts/check_examples_docs.sh --write

Config and Function Docs Check#

configs.md, aggregate_functions.md, scalar_functions.md, and window_functions.md are generated by dev/update_config_docs.sh and dev/update_function_docs.sh. To check that they are up to date, run the ci/scripts/check_generated_docs.sh script, which also runs as part of ./dev/rust_lint.sh. Run it with --write to update the pages.

Documentation HTML Build#

ci/scripts/check_docs_html.sh builds the documentation website, the same build the “Test doc build” job runs. Sphinx builds with -W, so one warning fails the check. ./dev/rust_lint.sh runs it. Run the script with --help for the tools it needs and the files it writes.

./ci/scripts/check_docs_html.sh

Benchmarks#

Criterion Benchmarks#

Criterion is a statistics-driven micro-benchmarking framework used by DataFusion for evaluating the performance of specific code-paths. In particular, the criterion benchmarks help to both guide optimisation efforts, and prevent performance regressions within DataFusion.

Criterion integrates with Cargo’s built-in benchmark support and a given benchmark can be run with

cargo bench --bench BENCHMARK_NAME

A full list of benchmarks can be found here.

cargo-criterion may also be used for more advanced reporting.

Microbenchmark health checks (local-only)#

Runs each microbenchmark case once to catch runtime panics, without measuring performance:

cargo xtask ci step test microbenchmarks

The suite took about 10 minutes in CI, so execution is local-only, including on release branches; no hosted workflow or required check is provided. Runtime failures may go unnoticed until a local run. When changing benchmark fixtures, setup, or APIs, run the affected benchmarks and include the commands and results in your PR.

Parquet SQL Benchmarks#

The parquet SQL benchmarks can be run with

 cargo bench --bench parquet_query_sql

These randomly generate a parquet file, and then benchmark queries sourced from parquet_query_sql.sql against it. This can therefore be a quick way to add coverage of particular query and/or data paths.

If the environment variable PARQUET_FILE is set, the benchmark will run queries against this file instead of a randomly generated one. This can be useful for performing multiple runs, potentially with different code, against the same source data, or for testing against a custom dataset.

The benchmark will automatically remove any generated parquet file on exit, however, if interrupted (e.g. by CTRL+C) it will not. This can be useful for analysing the particular file after the fact, or preserving it to use with PARQUET_FILE in subsequent runs.

Comparing Baselines#

By default, Criterion.rs will compare the measurements against the previous run (if any). Sometimes it’s useful to keep a set of measurements around for several runs. For example, you might want to make multiple changes to the code while comparing against the master branch. For this situation, Criterion.rs supports custom baselines.

 git checkout main
 cargo bench --bench sql_planner -- --save-baseline main
 git checkout YOUR_BRANCH
 cargo bench --bench sql_planner --  --baseline main

Note: For MacOS it may be required to run cargo bench with sudo

sudo cargo bench ...

More information on Baselines

Upstream Benchmark Suites#

Instructions and tooling for running upstream benchmark suites against DataFusion can be found in benchmarks.

These are valuable for comparative evaluation against alternative Arrow implementations and query engines.