Spark Benchmark Configuration#

How vanilla Spark 4.1.3 was configured for the TPC-H SF1000 results on the benchmarking page. The cluster, data and queries are described in Benchmark Environment.

Spark configuration (highlights)#

Both Spark runs use these settings. The vanilla Spark run has no Comet plugin and uses the stock SortShuffleManager.

Key

Value

spark.executor.instances

32

spark.executor.cores

16 (task parallelism per executor)

spark.kubernetes.executor.limit.cores / .request.cores

8 (physical vCPU)

spark.executor.memory

64G

spark.executor.memoryOverhead

10G

spark.memory.fraction

0.6

spark.memory.storageFraction

0.2

spark.sql.shuffle.partitions

512

spark.sql.broadcastTimeout

900

spark.serializer

KryoSerializer

spark.io.compression.codec

zstd

Spark AQE is left at its Spark 4.1 defaults. Shuffle spills to a gp3-backed per-executor volume (spark.kubernetes.executor.volumes...spark-local-dir-1).

Note that spark.executor.cores=16 is Spark’s task parallelism setting, not a CPU allocation — each executor pod is given only 8 physical vCPU via spark.kubernetes.executor.limit.cores / .request.cores, so Spark schedules 16 concurrent tasks onto 8 physical cores (2× oversubscription). The matching Ballista executor runs --vcores=8 on the same 8 physical vCPU (1:1).

Reproducing#

Until the harness is added, the closest public equivalent is tpcbench.py from apache/datafusion-benchmarks, with the settings above and stock Spark 4.1 defaults for everything else:

spark-submit \
  --master <master> \
  --conf spark.executor.instances=32 \
  --conf spark.executor.cores=16 \
  --conf spark.executor.memory=64G \
  --conf spark.executor.memoryOverhead=10G \
  --conf spark.sql.shuffle.partitions=512 \
  tpcbench.py \
    --benchmark tpch \
    --data s3a://<bucket>/tpch/sf1000 \
    --format parquet \
    --iterations 2