Memory Usage#

How much executor memory Ballista, Spark and Spark + Comet need to run one memory-intensive query, and how their speed changes as that memory shrinks. Executor memory is what a cluster is sized for: each executor reserves it for the whole job, so an engine that finishes in less memory runs on smaller or fewer machines.

These runs were made on a different, much smaller cluster than the TPC-H results, so their times are not comparable with that table.

How it was measured#

  • Query: TPC-H q18 at SF1000. Its subquery aggregates all of lineitem by order key, about 1.5 billion groups, and the result is joined with orders, customer and lineitem. It has the largest aggregation in TPC-H and large join inputs.

  • Cluster: Kubernetes on two servers with 32 CPU cores and 125 GiB of memory each, and 2 executors of 16 CPUs, one per server. The Parquet files were on each server’s local disk, not in S3.

  • Memory: the executor pod’s Kubernetes memory limit, which is everything the executor process may use. Each configuration was run at 64, 48, 32, 24, 16, 12 and 8 GiB until a size failed, and then at sizes in between, to within 2 GiB (1 GiB for Ballista below 8 GiB) of the smallest size that passed.

  • Passed means the query finished with no executor killed for running out of memory. Every size is a fresh cluster, and the query ran twice; times are the mean.

  • Versions: Ballista d9872080 with its default settings, Spark 4.1.3 with the settings in conf/spark.conf, and Comet 1.2.0 on the same Spark with conf/comet.conf.

How an executor divides its memory is a setting, and for Spark it changes the result. A Spark executor’s memory is its Java heap, optional off-heap memory and spark.executor.memoryOverhead for everything else the process allocates. Comet runs its native engine in the off-heap memory. Spark and Comet were each run with several divisions, given in the tables as shares of the executor’s memory. A Ballista executor gives 70% of its memory to the DataFusion memory pool that operators allocate from, and was run only with that default.

Results#

Bar chart of the smallest executor memory for each configuration

Line chart of q18 time against executor memory for each engine

Configuration

Memory split

Smallest size (GiB)

Mean there (s)

Largest size (GiB)

Mean there (s)

Ballista

memory pool 70% (default)

8

131.54

64

133.01

Spark, on-heap

heap 90%, overhead 10%

18

336.23

64

337.11

Spark, on-heap, 2x overhead

heap 80%, overhead 20%

12

406.50

24

331.87

Spark, 50% off-heap

heap 45%, off-heap 45%, overhead 10%

12

387.32

64

329.89

Spark + Comet, 50% off-heap

heap 45%, off-heap 45%, overhead 10%

24

159.75

64

148.50

Spark + Comet, 75% off-heap

heap 22.5%, off-heap 67.5%, overhead 10%

20

150.10

64

145.30

Spark + Comet, 75% off-heap, 2x overhead

heap 20%, off-heap 60%, overhead 20%

22

174.38

24

158.65

  • Ballista finished in the least memory, and its time stayed within a few seconds from 64 GiB down to its smallest size. The memory its executors held fell with the limit, so it ran the query in whatever memory it was given.

  • Spark’s smallest size was set by memoryOverhead, not the heap. At 16 GiB with all its memory on-heap, Spark’s executors went over their pod limit without the Java heap running out, and lost shuffle output then failed the query. Moving half of the heap off-heap, or doubling the overhead, let it finish in 12 GiB. Below about 24 GiB it slowed down.

  • Comet made Spark about twice as fast, but needed more memory than Spark. Every Comet failure was an executor killed for going over its pod limit, never an error from Comet’s memory pool, so Comet allocates memory that its pool does not account for. A larger off-heap share lowered its smallest size; a larger overhead did not.

  • Below its smallest size, Ballista did not fail cleanly. At 6 GiB executors were killed for running out of memory and restarted, and the query finished at about a third of its usual speed. At 5 and 4 GiB the query stopped making progress after the restarts instead of failing, and was stopped at the time limit.

Each size ran once, so a size close to an engine’s smallest could go either way on another run.

Reproducing#

Every point is one run of benchmarks/cluster/run.py with the size and division of memory given as flags. For example, Ballista and Spark with half its memory off-heap at 12 GiB:

cd benchmarks/cluster
COMMON="--query 18 --executors 2 --executor-cpus 16 --driver-cpus 4 --driver-memory-gib 16"
python3 run.py --engine ballista $COMMON --vcores 16 --executor-memory-gib 12
python3 run.py --engine spark $COMMON \
  --spark-conf spark.executor.cores=16 \
  --spark-conf spark.executor.memory=5529m \
  --spark-conf spark.memory.offHeap.enabled=true \
  --spark-conf spark.memory.offHeap.size=5530m \
  --spark-conf spark.executor.memoryOverhead=1229m

run.py reads the data from S3, which these runs did not, so expect different times. The results are in published/memory, and publish.py generates the tables and charts on this page from them.

All runs#

Configuration

Executor (GiB)

Heap / off-heap / overhead (GiB)

Result

Mean (s)

Iterations (s)

Ballista

64

memory pool 44.8

Passed

133.01

136.90, 129.11

48

memory pool 33.6

Passed

130.14

129.49, 130.80

32

memory pool 22.4

Passed

128.22

127.80, 128.65

24

memory pool 16.8

Passed

128.95

127.57, 130.34

16

memory pool 11.2

Passed

125.40

124.89, 125.91

12

memory pool 8.4

Passed

128.60

129.69, 127.53

8

memory pool 5.6

Passed

131.54

132.18, 130.90

6

memory pool 4.2

Passed after 2 executors ran out of memory

363.11

366.22, 360.00

5

memory pool 3.5

Did not finish within the time limit

4

memory pool 2.8

Did not finish within the time limit

Spark, on-heap

64

57.6 / 0.0 / 6.4

Passed

337.11

328.35, 345.87

48

43.2 / 0.0 / 4.8

Passed

341.81

339.37, 344.24

32

28.8 / 0.0 / 3.2

Passed

331.16

328.01, 334.32

24

21.6 / 0.0 / 2.4

Passed

329.85

322.26, 337.43

20

18.0 / 0.0 / 2.0

Passed

334.06

326.78, 341.33

18

16.2 / 0.0 / 1.8

Passed

336.23

330.08, 342.37

16

14.4 / 0.0 / 1.6

Failed: an executor ran out of memory

Spark, on-heap, 2x overhead

24

19.2 / 0.0 / 4.8

Passed

331.87

321.57, 342.17

20

16.0 / 0.0 / 4.0

Passed

339.06

333.09, 345.03

16

12.8 / 0.0 / 3.2

Passed

343.40

346.15, 340.65

12

9.6 / 0.0 / 2.4

Passed

406.50

393.29, 419.70

8

6.4 / 0.0 / 1.6

Passed after 4 executors ran out of memory

693.86

723.26, 664.45

Spark, 50% off-heap

64

28.8 / 28.8 / 6.4

Passed

329.89

331.98, 327.80

48

21.6 / 21.6 / 4.8

Passed

330.26

330.95, 329.58

32

14.4 / 14.4 / 3.2

Passed

334.27

335.93, 332.61

24

10.8 / 10.8 / 2.4

Passed

342.15

340.42, 343.89

16

7.2 / 7.2 / 1.6

Passed

353.43

359.04, 347.83

12

5.4 / 5.4 / 1.2

Passed

387.32

391.03, 383.61

10

4.5 / 4.5 / 1.0

Passed after 2 executors ran out of memory

501.25

600.58, 401.91

8

3.5 / 3.5 / 1.0

Failed: an executor ran out of memory

Spark + Comet, 50% off-heap

64

28.8 / 28.8 / 6.4

Passed

148.50

150.76, 146.24

48

21.6 / 21.6 / 4.8

Passed

153.68

155.36, 151.99

32

14.4 / 14.4 / 3.2

Passed

158.00

160.04, 155.95

24

10.8 / 10.8 / 2.4

Passed

159.75

163.86, 155.64

22

9.9 / 9.9 / 2.2

Failed: an executor ran out of memory

20

9.0 / 9.0 / 2.0

Failed: an executor ran out of memory

16

7.2 / 7.2 / 1.6

Failed: an executor ran out of memory

Spark + Comet, 75% off-heap

64

14.4 / 43.2 / 6.4

Passed

145.30

145.74, 144.86

48

10.8 / 32.4 / 4.8

Passed

149.09

150.66, 147.51

32

7.2 / 21.6 / 3.2

Passed

152.25

153.69, 150.81

24

5.4 / 16.2 / 2.4

Passed

153.26

156.09, 150.44

20

4.5 / 13.5 / 2.0

Passed

150.10

148.50, 151.69

18

4.0 / 12.2 / 1.8

Failed: an executor ran out of memory

16

3.6 / 10.8 / 1.6

Failed: an executor ran out of memory

Spark + Comet, 75% off-heap, 2x overhead

24

4.8 / 14.4 / 4.8

Passed

158.65

157.95, 159.35

22

4.4 / 13.2 / 4.4

Passed

174.38

155.53, 193.22

20

4.0 / 12.0 / 4.0

Failed: an executor ran out of memory