Spark Benchmark Configuration#

How vanilla Spark 4.1.3 was configured for the TPC-H SF1000 results on the benchmarking page. The cluster, data and queries are described in Benchmark Environment.

Spark configuration (highlights)#

Both Spark runs use these settings. The vanilla Spark run has no Comet plugin and uses the stock SortShuffleManager.

Key	Value
`spark.executor.instances`	`32`
`spark.executor.cores`	`16` (task parallelism per executor)
`spark.kubernetes.executor.limit.cores` / `.request.cores`	`8` (physical vCPU)
`spark.executor.memory`	`64G`
`spark.executor.memoryOverhead`	`10G`
`spark.memory.fraction`	`0.6`
`spark.memory.storageFraction`	`0.2`
`spark.sql.shuffle.partitions`	`512`
`spark.sql.broadcastTimeout`	`900`
`spark.serializer`	`KryoSerializer`
`spark.io.compression.codec`	`zstd`

Spark AQE is left at its Spark 4.1 defaults. Shuffle spills to a gp3-backed per-executor volume (spark.kubernetes.executor.volumes...spark-local-dir-1).

Note that spark.executor.cores=16 is Spark’s task parallelism setting, not a CPU allocation — each executor pod is given only 8 physical vCPU via spark.kubernetes.executor.limit.cores / .request.cores, so Spark schedules 16 concurrent tasks onto 8 physical cores (2× oversubscription). The matching Ballista executor runs --vcores=8 on the same 8 physical vCPU (1:1).

Reproducing#

With the scripts in benchmarks/cluster, after setting up cluster.toml as its README describes:

cd benchmarks/cluster
python3 run.py --engine spark

The settings above are in benchmarks/cluster/conf/spark.conf. The driver runs spark/tpcbench.py in client mode in a Kubernetes Job, and Spark starts the executors with a volume each for shuffle files, from the storage class in cluster.toml.