Comet Plugin Architecture#
Overview#
The Comet plugin enhances Spark SQL by introducing optimized query execution and shuffle mechanisms leveraging native code. It integrates with Spark’s plugin framework and extension API to replace or extend Spark’s default behavior.
Plugin Components#
Comet SQL Plugin#
The entry point to Comet is the org.apache.spark.CometPlugin class, which is registered in Spark using the following
configuration:
--conf spark.plugins=org.apache.spark.CometPlugin
The plugin has a driver component, CometDriverPlugin, and an executor component, CometExecutorPlugin.
CometDriverPlugin runs once, when the SparkContext starts and before any SparkSession exists, so it can set static
configuration that cannot be changed once a session has been created. It first sets spark.comet.version to the Comet
version. If neither spark.memory.offHeap.enabled nor spark.comet.exec.onHeap.enabled is true, it logs a warning
and skips the remaining steps. Otherwise it:
Appends
CometSparkSessionExtensionstospark.sql.extensions, unless it is already listed.On Spark 4.0 and later, sets
spark.shuffle.managerto Comet’sCometShuffleManager, whenspark.comet.enabledand Comet shuffle are on and the application has not setspark.shuffle.manager. An application’s own value is always kept. It is not set where Comet would disable itself anyway, such as on a big-endian platform, withspark.sql.parquet.int96TimestampConversion=true, or when the native library fails to load. Before Spark 4.0 the plugin cannot do this, because Spark has created the shuffle manager by the time plugins run. It runs before the cache serializer step, since that serializer is installed only with one of Comet’s shuffle managers.Sets
spark.sql.cache.serializerto Comet’sArrowCachedBatchSerializerwhenspark.comet.exec.inMemoryCache.enabled=true, unless the application has chosen a different serializer. It leaves Spark’s serializer in place where Comet could not scan its format in the Comet pipeline or Kryo would reject it; see In-Memory Cache.Registers
CometSourcewith Spark’s metrics system and addsCometMetricsListenertospark.sql.queryExecutionListenerswhenspark.comet.metrics.enabled=true.Logs a warning for settings that are likely to cause problems, such as an unset
spark.executor.memoryOverhead.
The plugin does not change any executor memory setting. The Memory Tuning guide covers how to size them.
When the driver or an executor stops, the plugin shuts down Comet’s tokio runtime in that JVM.
When spark.eventLog.enabled is true, the executor plugin also carries each sample of the executor’s native memory
usage log to the driver plugin. For each executor, the driver plugin keeps the sample with the most untracked memory
and the last sample of every minute, and posts them to Spark’s listener bus as CometExecutorMemoryUsage events so
that the event log records them. A listener on a listener bus queue of its own posts them at the executor’s first
heartbeat a minute or more after the first of them arrived, which also ends an idle executor’s minute, and posts what
the driver plugin holds when an executor is removed and when the application ends, since an executor that goes away
sends nothing more. See
Reading the Memory Usage Log from the Event Log.
CometSparkSessionExtensions can also be registered without the plugin, through spark.sql.extensions or
SparkSession.Builder.withExtensions. Most of Comet’s test suites and the Spark SQL tests enable Comet this way, so
none of the driver plugin’s steps run for them.
CometSparkSessionExtensions#
On initialization, this class registers Comet’s rules with Spark. The main one is CometRule, which converts the
plan. When Adaptive Query Execution is disabled, it runs once for the entire plan, before Spark inserts columnar
transitions. When Adaptive Query Execution is enabled, it runs as a query stage preparation rule, and again on
each new query stage before Spark inserts that stage’s transitions.
The class also registers rules that run after the transitions are inserted, such as
RevertNativeForTransitionHeavyStages and EliminateRedundantTransitions, Adaptive Query Execution stage optimizer
rules on Spark 3.5 and later, and the IcebergWriteStrategy planner strategy. The comment on
CometSparkSessionExtensions describes where each rule runs.
CometRule is two phases, applied in order: scan conversion (CometScanRule), then operator conversion
(CometExecRule). The order matters, because operator conversion builds its native plan up from the nodes that scan
conversion produces. Each phase is described below.
Phase 1: CometScanRule#
CometScanRule replaces any Parquet scans with Comet operators. There are different paths for Spark v1 and v2 data sources.
When reading from Parquet v1 data sources, Comet replaces FileSourceScanExec with a CometScanExec, and for v2
data sources, BatchScanExec is replaced with CometBatchScanExec. In both cases, Comet replaces Spark’s Parquet
reader with a custom vectorized Parquet reader. This is similar to Spark’s vectorized Parquet reader used by the v2
Parquet data source but leverages native code for decoding Parquet row groups directly into Arrow format.
Comet only supports a subset of data types and will fall back to Spark’s scan if unsupported types
exist. Comet can still accelerate the rest of the query execution in this case because CometSparkToColumnarExec will
convert the output from Spark’s scan to Arrow arrays. Note that both spark.comet.exec.enabled=true and
spark.comet.convert.parquet.enabled=true must be set to enable this conversion.
Refer to the Supported Spark Data Types section in the contributor guide to see a list of currently supported data types.
Phase 2: CometExecRule#
This rule traverses bottom-up from the original Spark plan and attempts to replace each operator with a Comet equivalent.
For example, a ProjectExec will be replaced by CometProjectExec.
When replacing a node, various checks are performed to determine if Comet can support the operator and its expressions. If an operator, expression, or data type is not supported by Comet then the reason will be stored in a tag on the underlying Spark node and that operator will not be converted.
Comet can replace part of a query stage. After CometRule runs, Spark inserts a ColumnarToRowExec where a
row-based operator reads columnar output, and Comet’s post-transition rules then run in order. First, when
spark.comet.exec.transitionRevert.enabled=true, RevertNativeForTransitionHeavyStages reverts a query stage to Spark
if it has more columnar-to-row transitions than spark.comet.exec.transitionRevert.maxTransitions, because the overhead
of the transitions can outweigh the benefits of running part of the stage in Comet. It keeps the stage in Comet when
reverting it is not safe, for example when Spark and Comet would have to exchange aggregate state. Then
EliminateRedundantTransitions removes transitions that are not needed and replaces the rest that read Comet output
with CometColumnarToRowExec. A schema that contains VariantType keeps Spark’s ColumnarToRowExec.
Query Execution#
Once the plan has been transformed, consecutive CometNativeExec operators are combined into a single block, which contains
a protocol buffer serialized version of the plan (the serialization code can be found in QueryPlanSerde). The write
operators are the exception: the child of CometWriteFilesExec, CometNativeWriteExec, or CometIcebergWriteExec
starts a block of its own.
Spark serializes the physical plan and sends it to the executors when executing tasks. The executors deserialize the plan and invoke it.
When CometNativeExec is invoked, it will pass the serialized protobuf plan into
Native.createPlan, which invokes the native code via JNI, where the plan is then deserialized.
In the native code there is a PhysicalPlanner struct (in planner.rs) which converts the deserialized plan into an
Apache DataFusion ExecutionPlan. In some cases, Comet provides specialized physical operators and expressions to
override the DataFusion versions to ensure compatibility with Apache Spark.
Not every expression in a Rust plan runs in Rust. When an expression has no Rust implementation, or its Rust
implementation does not match Spark for the given input, its serde can route it through the JVM codegen dispatcher
instead of falling back to Spark (see the
Implementation legend). The Rust plan then evaluates the
expression with JvmScalarUdfExpr, which passes each batch’s input columns through JNI and the Arrow C Data Interface
to CometUdfBridge, which hands them to CometScalaUDFCodegen on the JVM. There, Spark’s own generated code for the expression, compiled into a batch
kernel by CometBatchKernelCodegen, runs over the Arrow vectors and returns an Arrow column. Scala and Java UDFs take
the same path. The operator around the expression stays Rust-implemented, so this JVM code runs inside the Comet
pipeline. spark.comet.exec.scalaUDF.codegen.enabled, which is on by default, turns the dispatcher on. When it is off,
these expressions fall back to Spark.
A leaf node in the physical plan that takes Arrow batches from the JVM is a ScanExec. Each such input is exported once as an
Arrow C Stream (org.apache.arrow.c.ArrowArrayStream),
and ScanExec pulls each input batch through a single C callback rather than making a JNI call per batch. The
input could be a broadcast, a shuffle read decoded through JNI and imported into the JVM, the output of
CometSparkToColumnarExec, or the output of an earlier Rust plan. A shuffle input is a ShuffleScanExec instead
only when it is serialized as a ShuffleScan leaf: direct shuffle read is enabled, Adaptive Query Execution has made
the Comet shuffle a query stage, and Comet supports its output types. ShuffleScanExec pulls compressed Arrow IPC
blocks from the JVM and decodes them in Rust (see Shuffle Reads). Otherwise, including without
Adaptive Query Execution and for operators that run their own Rust plans such as CometTakeOrderedAndProjectExec, the
shuffle is read as batches through ScanExec. The Parquet, Iceberg, and CSV scans read their files in Rust.
CometNativeExec creates a CometExecIterator and applies this iterator to the input RDD
partitions. Each call to CometExecIterator.next() will invoke Native.executePlan. Once the plan finishes
executing, the resulting Arrow batches are imported into the JVM using Arrow FFI.
Shuffle#
Comet integrates with Spark’s shuffle mechanism, optimizing both shuffle writes and reads. Comet’s shuffle manager
must be registered with Spark using the following configuration. On Spark 4.0 and later, CometDriverPlugin
registers it when the application leaves spark.shuffle.manager unset (see above); on earlier versions, the
application must set it:
--conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager
Shuffle Writes#
For shuffle writes, a ShuffleMapTask runs in the executors. This task contains a ShuffleDependency that is
broadcast to all of the executors. It then passes the input RDD to ShuffleWriteProcessor.write() which
requests a ShuffleWriter from the shuffle manager, and this is where it gets a Comet shuffle writer.
Comet has two shuffle implementations, native shuffle and JVM columnar shuffle. When Native Shuffle is Used describes how Comet chooses between them.
For native shuffle, CometNativeShuffleWriter runs one native plan per task, with a ShuffleWriter operator at the
root. When the exchange’s child is a CometNativeExec subtree, that subtree becomes the writer’s child, so the operators
that produce the shuffle input and the writer run in the same native plan, and no batch crosses into the JVM between
them. Otherwise, for example when the exchange’s child is CometSparkToColumnarExec, the writer’s child is a scan that
reads batches from the JVM. The writer partitions the batches and writes them in Arrow IPC format. See
Native Shuffle for details.
JVM columnar shuffle takes rows instead, converting a Comet child’s output with ColumnarToRowExec. It assigns
partitions with Spark’s partitioner, buffers the rows in memory pages, and calls native code to encode them to Arrow
IPC. See JVM Shuffle for details.
Shuffle Reads#
For shuffle reads, CometShuffledBatchRDD requests a ShuffleReader from the shuffle manager and gets a
CometBlockStoreShuffleReader, which fetches blocks with Spark’s ShuffleBlockFetcherIterator. Both shuffle
implementations write the same Arrow IPC block format, so the same reader serves both. When a Rust plan reads the
shuffle output as a ShuffleScan input (see Query Execution) and
spark.comet.shuffle.directRead.enabled is true, the default, the compressed IPC blocks go to that plan,
which decodes them. Otherwise NativeBatchDecoderIterator decodes each block in native code through JNI, and Arrow
FFI imports the result into the JVM as a ColumnarBatch. See
Read Path for details.