Ballista Development#
We welcome participation from everyone and encourage you to join us, ask questions, and get involved.
All participation in the Apache DataFusion Ballista project is governed by the Apache Software Foundation’s code of conduct.
Development Environment#
The easiest way to get started if you are using VSCode or IntelliJ IDEA is to open the provided Dev Container
which will install all the required dependencies including Rust, protoc, Docker, and Node.js. A Dev Container is a
development environment that runs in a Docker container. It is configured with all the required dependencies to
build and test the project. Other supporting tools that use Dev Containers can be seen
here
To use the Dev Container, open the project in VS Code and then click the “Reopen in Container” button in the bottom right corner of the IDE.
If you are not using the Dev Container or VScode, you will need to install these dependencies yourself.
Protobuf Compiler is required to build the project.
Docker is required to run the integration tests.
Node.js is only needed to run
npx prettierover the Markdown docs.The
wasm32-unknown-unknowntarget and Trunk are only needed to build the Web TUI; see the Ballista CLI guide.
Build the project#
From the root of the project, build release binaries.
cargo build --release
Testing the project#
cargo test
Test fixtures#
Look for an existing fixture before writing one, and put a new one in the narrowest place that every test using it can reach:
Used by the tests of one module: in that module’s tests.
Shared by the unit tests of several modules of a crate: in that crate’s
#[cfg(test)]test utilities module.Shared by the integration tests of a crate: in that crate’s
tests/common/mod.rs.Needed by other crates, such as the TPC-H table schemas: in
ballista_core::test_utils.
A crate’s own unit tests cannot use a helper crate that depends on that crate, because they would see two copies of it, so unit-test fixtures stay inside their crate.
Tests that need table data write it into a tempfile::TempDir rather than add files to the repository. Data
that has to be checked in lives in the testdata directory of the crate that reads it.
TPC-H plan-stability golden files#
The tpch_plan_stability test suite in the ballista-scheduler crate guards the shape of the
distributed physical plans that Ballista produces for the 22 TPC-H queries. For each query it plans
the SQL and compares the staged plan text — the stage boundaries (=== Stage N === banners),
ShuffleWriterExec / UnresolvedShuffleExec nodes, and any exchange reuse — against an approved
“golden” file. The queries are planned against a synthetic statistics-only table, so the suite is
deterministic and runs without TPC-H data or a running cluster.
The golden files live in
ballista/scheduler/tests/tpch_plan_stability/approved/
(q1.txt through q22.txt). When a change alters distributed planning, the suite fails with a
plan-drift diff. Review the diff: if the change is unintended, fix the regression; if the new plans
are correct, regenerate the golden files:
./dev/update-tpch-plan-stability.sh
This runs the suite in generate mode
(BALLISTA_GENERATE_GOLDEN=1 cargo test -p ballista-scheduler --test tpch_plan_stability) and
rewrites the approved files. Review the resulting changes under
ballista/scheduler/tests/tpch_plan_stability/approved/ and commit them together with the code
change that caused them.
Running the examples#
cd examples
cargo run --example standalone_sql --features=ballista/standalone
Building the Python Client from Source#
The Python client (ballista on PyPI) lives in the python/ directory and is versioned and released
independently from the main Ballista project, so it is intentionally not part of the default Cargo
workspace. Building it from source uses maturin to compile the Rust
extension module and install it into a Python virtual environment.
Prerequisites#
Python 3.10 or newer
The Rust toolchain (see Development Environment)
protocon yourPATH
All commands in this section are run from the python/ directory:
cd python
Create a virtual environment#
Using pip:
python3 -m venv .venv
source .venv/bin/activate
pip3 install --upgrade pip
pip3 install -r requirements.txt
The pip upgrade is required because maturin develop invokes pip install --group, which
needs pip 25.1 or newer. The pip shipped with python3 -m venv is often older than that.
Using uv:
uv sync --dev --no-install-package ballista
Build and install#
maturin develop compiles the Rust extension and installs it into the active virtual environment
as an editable package, so subsequent Python imports pick up your local changes.
Using pip:
maturin develop # debug build
maturin develop --release # release build (slower to compile, faster at runtime)
Using uv:
uv run --no-project maturin develop --uv
To produce a wheel without installing it (for example, to share or publish):
uv run --no-project maturin build --release --strip
Run the tests#
Using pip:
python3 -m pytest
Using uv:
uv run --no-project pytest
For more detail on the underlying workflow, including tips for improving build speed, see the DataFusion Python contributor guide.
Benchmarking#
For performance testing and benchmarking with TPC-H and other datasets, see the benchmarks README.
This includes instructions for:
Generating TPC-H test data
Running benchmarks against DataFusion and Ballista
Comparing performance with Apache Spark
Running load tests
Merge queue and the TPC suites in CI#
Pull requests to main are merged through GitHub’s merge queue. The queue builds the exact tree that will land
and only merges it if the required checks pass, so a PR does not need to be rebased and re-tested by hand.
Use “Merge when ready” (auto-merge) to queue a PR before its checks have finished.
The TPC-H SF10 and TPC-DS SF1 suites are slow, so they run in the queue rather than on every PR:
Event |
TPC-H / TPC-DS |
|---|---|
Pull request |
Only if the PR has the |
Merge queue |
Both, whenever the change touches code or build files (docs-only changes skip) |
Push to |
Not run; the queue already tested that tree. Only the build caches are refreshed |
Manual |
|
Apply a label to get early feedback on a change that could affect query results or plans, such as planner, shuffle or scheduler work. Applying the label runs the suite immediately without blocking anything. The label also makes the suite run on every later push, and those runs do block the PR. If a suite fails in the queue the PR is removed from it, so a failure there is worth reproducing with the label before re-queueing.
The single required status check is TPC Required Checks from .github/workflows/tpc.yml.
DataFusion dependency#
Between releases, main depends on a pinned git revision of
apache/datafusion main rather than a crates.io release, so
that breaking changes arrive in small increments instead of piling up at release time.
The datafusion* entries in the workspace Cargo.toml move through one cycle per release:
Phase |
|
|---|---|
Between releases |
pinned git |
DataFusion release candidate |
the RC tag, to validate early |
DataFusion released |
the crates.io version |
Ballista release branch cut |
(branch keeps the crates.io version) |
Back on |
pinned git |
A release branch therefore only ever depends on published crates. Each rev bump on main is a PR
that ports any DataFusion API changes, refreshes affected golden files, and keeps cargo test and
cargo clippy --all-targets --workspace -- -D warnings green.
The Python client in python/ is a separate Cargo workspace that also depends on
datafusion-python. It can only move to a new
DataFusion major version once datafusion-python has published one, usually some weeks after
DataFusion, so it is released separately from the Rust crates. See the
Python client release process.
The remaining release steps are described in the release process.
Backporting to a release branch#
Each release branch, such as branch-55, also makes minor and patch releases, so a fix or a
backward-compatible feature that has merged to main can reach users before the next major version.
See Versioning and Compatibility for what can be released this way.
A merged change can be backported if all of these hold:
It does not need an entry in the upgrade guide. It does not break a public API, rename or remove a configuration key or CLI flag, or change a default that alters query plans or results.
It does not carry the
api-changelabel.It does not change
BALLISTA_PROTOCOL_VERSION, and it does not change the wire format in a way that an older scheduler, executor or client would misread. PRs that change the wire format carry theproto-changelabel, so check those against this rule.
Backport a change when it merges, not in a batch before a release. Conflicts are cheapest to resolve
while the code is fresh, and main moves further from the release branch every week. The author of a
change is best placed to backport it, and a reviewer can ask for a backport.
Create a branch from the release branch, for example
git switch --no-track -c backport-2062 apache/branch-55.Cherry-pick the commit from
mainwithgit cherry-pick -x <commit>, which records the original commit in the message.Open a PR against the release branch. Title it with the original title, the branch in brackets and the original PR number, and copy the labels from the original PR. The
54.1.0release, for example, includedfix: [branch-54] keep EmptyExec partition count intact across stage serialization (backport of #2062).
A backport goes through the same review and CI as any other PR.
Upgrade notes#
When a change breaks a public API, renames or removes a configuration key or CLI
flag, changes a default that alters query plans or results, or changes the
wire/serialization format, add an entry describing the change and the required
action to the current unreleased page under
docs/source/upgrading/. This keeps the upgrade guide
complete for the next release.
Add the api-change label to the PR as well when the change breaks a public API. A change that needs
an upgrade note cannot be released in a minor or patch release, so this also decides whether it can be
backported to a release branch.
Add the proto-change label when the change alters the wire or serialization format, for example by
editing a .proto file. Such a change can still be backported if older components can safely ignore it,
such as a new field that they never read.