Spark + Comet Benchmark Configuration

How Spark 4.1.3 with Apache DataFusion Comet 1.1.0-rc2 was configured for the TPC-H SF1000 results on the benchmarking page. The cluster, data and queries are described in Benchmark Environment.

Comet configuration

The Spark + Comet run adds these settings to the Spark configuration:

Key	Value
spark.plugins	org.apache.spark.CometPlugin
spark.comet.enabled	true
spark.shuffle.manager	org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager
spark.memory.offHeap.enabled	true
spark.memory.offHeap.size	32G
spark.comet.exec.memoryPool.fraction	0.8

The run also enabled Comet’s explain and fallback logging. Everything else is left at Comet’s defaults.

Reproducing

Build the Comet image from benchmarks/cluster/docker/comet.Dockerfile, which adds the Comet JAR to the Spark image (see the Comet installation guide), then run:

cd benchmarks/cluster
python3 run.py --engine comet

The settings above are in benchmarks/cluster/conf/comet.conf, on top of the Spark settings.