Ballista Benchmark Configuration#

How Ballista was configured for the TPC-H SF1000 results on the benchmarking page. The cluster, data and queries are described in Benchmark Environment.

Ballista configuration#

Flag / config key

Value

--vcores

8

--memory-pool-size (bytes; ≈70 % of the 64 GiB container)

48103633715

--work-dir

/data (dedicated gp3 PVC — see Executor storage)

--grpc-server-max-decoding-message-size

134217728

--grpc-server-max-encoding-message-size

134217728

datafusion.execution.target_partitions

256

datafusion.execution.collect_statistics

true

ballista.planner.adaptive.enabled

true (AQE; default)

ballista.shuffle.sort_based.memory_limit_per_task_bytes

268435456 (256 MiB; default)

datafusion.optimizer.prefer_hash_join is left at its default; under AQE the join strategy is selected at runtime by DelayJoinSelectionRule / DynamicJoinSelectionExec from runtime statistics and the broadcast / ballista.optimizer.hash_join_max_build_partition_bytes thresholds.

The gRPC message-size ceiling is raised from the 16 MiB default to 128 MiB; some SF1000 physical plans (Q11, Q21, Q22) encode above 16 MiB and hit OutOfRange errors otherwise.

Reproducing#

Bring up the cluster (one scheduler, N executors), then run the suite from a client. Executor sizing on each node:

ballista-executor \
  --bind-host 0.0.0.0 --bind-port 50051 \
  --scheduler-host <scheduler> --scheduler-port 50050 \
  --vcores 8 \
  --memory-pool-size 48103633715 \
  --work-dir /data \
  --grpc-server-max-decoding-message-size 134217728 \
  --grpc-server-max-encoding-message-size 134217728

/data should be a dedicated volume (e.g. a gp3 PVC) sized for the suite’s shuffle output — see Executor storage.

Run all 22 queries with the tpch Rust runner from benchmarks/:

cargo run --release --bin tpch -- benchmark ballista \
  --host <scheduler> --port 50050 \
  --path s3://<bucket>/tpch/sf1000 --format parquet \
  --partitions 256 --iterations 2

The runner sets target_partitions from --partitions and enables collect_statistics; everything else is left at its default.