JVM Shuffle#
This document describes Comet’s JVM-based columnar shuffle implementation (CometColumnarShuffle), which
writes shuffle data in Arrow IPC format. Partition assignment runs on the JVM and Arrow IPC encoding runs in
Rust. For the alternative that partitions in Rust, see Native Shuffle.
Overview#
Comet provides two shuffle implementations:
CometNativeShuffle (
CometExchange): Shuffle implemented in Rust. Takes Arrow batches directly from Comet operators and performs partitioning in Rust.CometColumnarShuffle (
CometColumnarExchange): JVM-based shuffle that operates on rows internally, buffersUnsafeRows in memory pages, and uses native code (via JNI) to encode them to Arrow IPC format. Uses Spark’s partitioner for partition assignment. Can accept either row-based or columnar input (columnar input is converted to rows viaColumnarToRowExec).
The JVM shuffle is selected via CometShuffleDependency.shuffleType.
When JVM Shuffle is Used#
JVM shuffle (CometColumnarExchange) is used instead of native shuffle (CometExchange) in the following cases:
Shuffle mode is explicitly set to “jvm”: When
spark.comet.shuffle.modeis set tojvm.Child plan is not a Comet operator: When the child plan is a Spark row-based operator (not a
CometPlan), JVM shuffle is used, since native shuffle requires columnar input from Comet operators. The exception isspark.comet.convert.shuffleInput.enabled, which converts the child’s rows to Arrow withCometSparkToColumnarExecso that native shuffle can take the shuffle instead. See When Native Shuffle is Used.Unsupported partition key types:
RangePartitioningkeys must be primitive, so a complex range key always falls back here.HashPartitioningkeys must be primitive only by default: settingspark.comet.shuffle.native.partitioning.hash.nested.enabledtotruekeeps struct and array keys, and map keys on Spark 4.0 and later, with native shuffle. The config defaults tofalse, so a complex hash key falls back to JVM columnar shuffle unless it is enabled. See Supported partition key types for the exact rules. Complex types are fully supported as data columns in both implementations.Partition keys native shuffle cannot serialize: native shuffle serializes the partitioning expressions to protobuf, so a key expression Comet has no serde for, or whose serde reports it incompatible, keeps the exchange off the native path. JVM shuffle has no such requirement, because it evaluates the key on the JVM through
UnsafeProjectionandLazilyGeneratedOrdering, so these exchanges land here rather than on Spark’s shuffle. One example is themapsort(...)wrapper Spark 4.0 and later adds around a map used as a shuffle key: Comet cannot serialize it for array or struct map keys, so such an exchange becomesCometColumnarExchange.
Collated strings are the exception. A hash or range key whose type is a non-UTF8_BINARY
collated string is declined by both Comet paths, so the exchange stays a plain Spark
Exchange. CometCollationSuite treats that fallback as the primary line of defense for
collated keys and pins the reasons it emits. The operators above the exchange decline collated
keys independently, so this is one guard among several rather than the only one.
Input Handling#
Spark Row-Based Input#
When the child plan is a Spark row-based operator, CometColumnarExchange calls child.execute() which
returns an RDD[InternalRow]. The rows flow directly to the JVM shuffle writers.
Comet Columnar Input#
When the child plan is a Comet operator that produces Arrow batches (e.g., CometHashAggregate) but
JVM shuffle is selected (due to shuffle mode setting or unsupported partitioning),
CometColumnarExchange still calls child.execute(). Comet operators that extend CometExec
implement doExecute() by wrapping themselves with ColumnarToRowExec:
// In CometExec base class
override def doExecute(): RDD[InternalRow] =
ColumnarToRowExec(this).doExecute()
This means the data path becomes:
Comet operator (Arrow batches) → ColumnarToRowExec → rows → JVM Shuffle → Arrow IPC → columnar
CometSparkToColumnarExec is different: its doExecute() returns its child’s rows directly, so a
JVM shuffle over it reads those rows without any Arrow conversion.
For a child that produces Arrow batches, this is less efficient than native shuffle, which avoids the columnar-to-row conversion:
Comet operator (Arrow batches) → Native Shuffle → Arrow IPC → columnar
Why Use Spark’s Partitioner?#
JVM shuffle uses row-based input so it can leverage Spark’s existing partitioner infrastructure
(partitioner.getPartition(key)). This allows Comet to support all of Spark’s partitioning schemes
without reimplementing them in Rust. Native shuffle, by contrast, serializes the partitioning scheme
to protobuf and implements the partitioning logic in Rust.
Architecture#
┌─────────────────────────────────────────────────────────────────────────┐
│ CometShuffleManager │
│ - Extends Spark's ShuffleManager │
│ - Routes to appropriate writer/reader based on ShuffleHandle type │
└─────────────────────────────────────────────────────────────────────────┘
│
┌────────────────────────┼────────────────────────┐
▼ ▼ ▼
┌─────────────────────┐ ┌─────────────────────┐ ┌─────────────────────┐
│ CometBypassMerge- │ │ CometUnsafe- │ │ CometNative- │
│ SortShuffleWriter │ │ ShuffleWriter │ │ ShuffleWriter │
│ (hash-based) │ │ (sort-based) │ │ (writes in Rust) │
└─────────────────────┘ └─────────────────────┘ └─────────────────────┘
│ │
▼ ▼
┌─────────────────────┐ ┌─────────────────────┐
│ CometDiskBlock- │ │ CometShuffleExternal│
│ Writer │ │ Sorter │
└─────────────────────┘ └─────────────────────┘
│ │
└────────────┬───────────┘
▼
┌─────────────────────┐
│ SpillWriter │
│ (Rust encoding │
│ via JNI) │
└─────────────────────┘
Key Classes#
Shuffle Manager#
Class |
Location |
Description |
|---|---|---|
|
|
Entry point. Extends Spark’s |
|
|
Extends |
Shuffle Handles#
Handle |
Writer Strategy |
|---|---|
|
Hash-based: one file per partition, merged at end |
|
Sort-based: records sorted by partition ID, single output |
|
Native shuffle |
Selection logic in CometShuffleManager.shouldBypassMergeSort():
Uses bypass if partitions ≤ threshold AND partitions × cores ≤ max threads
Otherwise uses sort-based to avoid OOM from many concurrent writers
Writers#
Class |
Location |
Description |
|---|---|---|
|
|
Hash-based writer. Creates one |
|
|
Sort-based writer. Uses |
|
|
Buffers rows in memory pages for a single partition. Spills to disk via Arrow IPC encoding in Rust. Used by bypass writer. |
|
|
Buffers records across all partitions, sorts by partition ID, spills sorted data. Used by unsafe writer. |
|
|
Holds one spill’s records for |
|
|
Base class for spill logic. Manages memory pages. |
Reader#
Class |
Location |
Description |
|---|---|---|
|
|
Fetches shuffle blocks via |
|
|
Reads compressed Arrow IPC batches from input stream. Calls |
Data Flow#
Write Path#
ShuffleWriteProcessorcallsCometShuffleManager.getWriter()Writer receives
Iterator[Product2[K, V]]where V isUnsafeRowRows are serialized and buffered in off-heap memory pages
When memory threshold or batch size is reached, the buffered rows are written.
CometDiskBlockWritercallsSpillWriter.doSpilling()for its partition.CometShuffleExternalSortercallsSpillSorter.writeSortedPartitions()once for each spill and once more for the final file, and each call writes every partitionNative code converts the rows to Arrow arrays and writes them as Arrow IPC blocks.
doSpilling()passes one partition’s row addresses and sizes toNative.writeSortedFileNative().SpillSortersorts its record pointers by partition withNative.sortRowPartitionsNative(), a radix sort that uses the half of the pointer arrayShuffleInMemorySorterkeeps free as scratch space, then passes them, with the address of each data page, toNative.writeSortedPartitionsNative(). That call reads the rows in place, writes every partition of the spill, and returns each partition’s length and checksumFor bypass writer: partition files are concatenated into final output
For sort writer: spill files are merged
Read Path#
CometBlockStoreShuffleReader.read()createsShuffleBlockFetcherIteratorFor each block,
NativeBatchDecoderIteratorreads the IPC streamNative code (
Native.decodeShuffleBlock()) decompresses and decodes to Arrow arraysArrow FFI imports arrays as
ColumnarBatch
This is the path taken when the consumer is a Spark operator. When a Rust plan reads the shuffle output as a
ShuffleScan input and spark.comet.shuffle.directRead.enabled is set, steps 2 through 4 are skipped:
the raw compressed blocks are handed to native code, which decodes them inside the plan. JVM shuffle
writes the same Arrow IPC block format as native shuffle, so direct read applies to both. See
Direct Read for the mechanism.
Memory Management#
CometShuffleMemoryAllocator.getInstancereturns the allocator for the task. Pages are alwaysUnsafe-allocated, because their addresses are handed to native code, but what bounds them depends on Spark’s memory mode.Off-heap mode gets
CometUnifiedShuffleMemoryAllocator, an ordinary SparkMemoryConsumerdrawing fromspark.memory.offHeap.size. When it cannot acquire a page the writer spills to disk. An allocation that loses the task’s entry in Spark’s execution pool while it waits (SPARK-59444) is retried a few times before it counts as a page that could not be acquired.On-heap mode gets
CometUnboundedShuffleMemoryAllocator, which keeps no budget and so never refuses a page. Nothing bounds these allocations, and memory pressure never triggers a spill. That mode exists only so the Spark SQL tests can run against Comet. See Memory Management.Row count still triggers spilling in either mode.
CometDiskBlockWriterspills atmin(spark.comet.shuffle.jvm.spillThreshold, batch size), where the batch size isspark.comet.shuffle.jvm.batchSizecapped atspark.comet.batchSize.CometShuffleExternalSorterspills atspark.comet.shuffle.jvm.spillThresholdalone, which defaults toInt.MaxValue, so on the sort path that trigger is effectively off by default.CometDiskBlockWritercoordinates spilling across all partition writers (largest first)The bypass writer’s partition writers each hold a page as soon as they buffer a row. Each one starts a batch on a 64 KiB page and doubles each further page, up to Spark’s page size (
spark.buffer.pageSize, capped at 128 MB). Spark sizes its page so that a task fits only 16 to 32 of them, so pages of that size would make the writers spill each other about one row at a time once a shuffle has more partitions than that. The sort-based writer has a single sorter per task, and uses Spark’s page size from the first page.
Configuration#
Config |
Description |
|---|---|
|
Rows per Arrow batch |
|
Row count threshold for spill |
|
Compression codec, |
|
Store offsets as lengths in zstd blocks without dictionary-encoded columns, |
|
Decode blocks inside the Rust plan on read, bypassing Arrow FFI, |