Spark + Comet Benchmark Configuration#

How Spark 4.1.3 with Apache DataFusion Comet 1.1.0-rc2 was configured for the TPC-H SF1000 results on the benchmarking page. The cluster, data and queries are described in Benchmark Environment.

Comet configuration#

The Spark + Comet run adds these settings to the Spark configuration:

Key	Value
`spark.plugins`	`org.apache.spark.CometPlugin`
`spark.comet.enabled`	`true`
`spark.shuffle.manager`	`org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager`
`spark.memory.offHeap.enabled`	`true`
`spark.memory.offHeap.size`	`32G`
`spark.comet.exec.memoryPool.fraction`	`0.8`

The run also enabled Comet’s explain and fallback logging. Everything else is left at Comet’s defaults.

Reproducing#

Start from the Spark spark-submit command, add the Comet JAR to the driver and executor classpath (see the Comet installation guide) and add the settings above:

  --conf spark.plugins=org.apache.spark.CometPlugin \
  --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \
  --conf spark.memory.offHeap.enabled=true \
  --conf spark.memory.offHeap.size=32G \
  --conf spark.comet.exec.memoryPool.fraction=0.8