Comet Tuning Guide#

Comet provides some tuning options to help you get the best performance from your queries. Every deployment needs to configure how much memory Comet can use, so start with memory tuning. The guide is split into the following pages:

  • Memory Tuning: configuring Comet’s off-heap memory pool and the executor memory overhead, choosing a memory pool, batch size, and limiting spill disk usage.

  • Shuffle Tuning: enabling Comet shuffle, the native and columnar shuffle implementations, limiting the native shuffle writer’s memory, and shuffle compression.

  • Remote Shuffle with Celeborn: bounding frame sizes and memory admission for native remote shuffle.

  • Parquet Scan Tuning: row filter pushdown and late materialization, and sizing splits so that row groups spread evenly across scan tasks.

  • Iceberg Scan Tuning: reading data files in parallel with spark.comet.scan.icebergNative.dataFileConcurrencyLimit.

  • Operator Tuning: joins, adaptive partial aggregation, and sorting on floating-point values.

  • Reducing Row/Columnar Conversion Overhead: stages in which many operators fall back to Spark, and the experimental direct columnar-to-row converter.

Nested and Wide Data#

Rows with many columns, long strings, or nested columns such as arrays of structs make every batch larger, so queries over such data spill more. For these workloads:

  • Choose the codec for spill files from native sorts, aggregations, and joins, which Comet compresses with lz4 by default. See Compressing Spill Files.

  • Lower the batch size if wide batches cause spilling or out-of-memory errors. See Batch Size.

  • Check which shuffles can run natively when partitioning keys are nested. See Native Shuffle.

  • Watch for row/columnar conversions, which are expensive for such schemas. See Wide or Deeply Nested Schemas.

Configuring Tokio Runtime#

Comet uses a global tokio runtime per executor process. By default it starts one worker thread per core that the executor runs tasks on, and allows up to 512 blocking threads, which is tokio’s default. Comet takes the number of cores from the first of these that applies:

  • the thread count of a local, local[N] or local[*] master, in local mode

  • spark.executor.cores, when it is set

  • the cores per worker C of a local-cluster[N, C, M] master

  • the number of processors available to the executor’s JVM on a standalone (spark://) cluster, since a standalone executor without spark.executor.cores takes every core its worker offers, and a worker offers all of its machine’s cores unless it is started with a different number (SPARK_WORKER_CORES)

  • one on YARN and Kubernetes, which is their default for spark.executor.cores

On any other cluster manager, such as Mesos, Comet cannot tell how many cores the executor has when spark.executor.cores is not set. It starts one worker thread for each processor available to the executor’s JVM, and logs a warning unless COMET_WORKER_THREADS is set. Too few worker threads cost more than a few idle ones: native plans that read no input from the JVM, such as a native Parquet scan feeding a sort, run entirely on the worker threads, so tasks share them when an executor runs more tasks at once than it has worker threads. The worker and blocking thread counts can be overridden using the environment variables COMET_WORKER_THREADS and COMET_MAX_BLOCKING_THREADS.

Metrics Overhead#

The SQL metrics described in Metrics are always collected. Setting spark.comet.metrics.enabled=true additionally publishes plan-coverage counters (operators.native, operators.spark, queries.planned, transitions, and acceleration.ratio) through Spark’s metrics system under the comet source. It is disabled by default because it walks every executed plan on the driver after each query, and the counters are only useful with an external sink (for example Prometheus) configured. This setting must be applied before the SparkSession is created.

Explain Plan#

For an explanation of Comet plan output, the configs that control it, and how fallback to Spark works, see Understanding Comet Plans.