Aggregate Expressions#

ApproximatePercentile#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • The percentage argument must be a foldable literal.

  • The accuracy argument must be a foldable literal.

  • Only byte, short, int, long, float, and double input types are supported.

Average#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • YearMonthIntervalType and DayTimeIntervalType inputs are not supported

CollectSet#

The following incompatibilities cause CollectSet to fall back to Spark by default. Set spark.comet.expression.CollectSet.allowIncompatible=true to enable Comet acceleration despite these differences.

  • Before Spark 4.2, Comet deduplicates NaN values (treats NaN == NaN) while Spark treats each NaN as a distinct value. Comet treats -0.0 and 0.0 as distinct while Spark treats them as equal. When spark.comet.exec.strictFloatingPoint=true, collect_set on floating-point types falls back to Spark on those versions unless spark.comet.expression.CollectSet.allowIncompatible=true is set.

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • collect_set over FLOAT or DOUBLE in a PartialMerge chain (for example, beside count(DISTINCT ...)) falls back to Spark to preserve signed-zero and NaN semantics.

First#

The following differences from Spark are always present and do not require any additional configuration:

  • This function is not deterministic. Results may not match Spark.

HllSketchAgg#

The following incompatibilities cause HllSketchAgg to fall back to Spark by default. Set spark.comet.expression.HllSketchAgg.allowIncompatible=true to enable Comet acceleration despite these differences.

  • Comet uses a Rust DataSketches port; HLL sketch bytes and estimates may differ slightly from Spark.

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • The lgConfigK argument must be a foldable literal.

  • The lgConfigK argument must be in the range [4, 21].

  • Only int, long, string, and binary input types are supported.

HllUnionAgg#

The following incompatibilities cause HllUnionAgg to fall back to Spark by default. Set spark.comet.expression.HllUnionAgg.allowIncompatible=true to enable Comet acceleration despite these differences.

  • Comet uses a Rust DataSketches port; HLL sketch bytes and estimates may differ slightly from Spark.

  • Errors surface as plain Comet execution errors rather than Spark’s SparkRuntimeException with condition HLL_UNION_DIFFERENT_LG_K / HLL_INVALID_INPUT_SKETCH_BUFFER (sqlState 22000), so the message and error class differ even though both engines fail.

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • The allowDifferentLgConfigK argument must be a foldable literal.

HyperLogLogPlusPlus#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • Input type must be boolean, integral, floating-point, decimal, date/time, string, or binary

  • DecimalType with precision > 18 is not supported

  • Collated (non-UTF8_BINARY) strings are not supported

Last#

The following differences from Spark are always present and do not require any additional configuration:

  • This function is not deterministic. Results may not match Spark.

ListAgg#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • BinaryType inputs are not supported.

  • WITHIN GROUP (ORDER BY ...) is not supported.

  • Non-literal delimiters are not supported.

  • Non-default string collations are not supported.

Max#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • Input of a type other than boolean, integral, floating-point, decimal, date, timestamp, string or binary, such as timestamp without time zone, intervals, arrays and maps. A struct is supported only if all of its fields are of supported types other than float and double, since Comet orders -0.0 and NaN with the sign bit set differently from Spark inside a struct.

  • A string with a non-default collation (Spark 4.0+), which Spark orders by its collation while Comet compares raw bytes.

MaxBy#

The following differences from Spark are always present and do not require any additional configuration:

  • This function is non-deterministic when multiple rows share the maximum ordering value. Results may differ from Spark in that case.

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • The value and ordering must both be fixed-length types (boolean, integral, floating-point, decimal, date, or timestamp). A variable-length or nested type such as string, binary, or struct falls back to Spark.

Min#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • Input of a type other than boolean, integral, floating-point, decimal, date, timestamp, string or binary, such as timestamp without time zone, intervals, arrays and maps. A struct is supported only if all of its fields are of supported types other than float and double, since Comet orders -0.0 and NaN with the sign bit set differently from Spark inside a struct.

  • A string with a non-default collation (Spark 4.0+), which Spark orders by its collation while Comet compares raw bytes.

MinBy#

The following differences from Spark are always present and do not require any additional configuration:

  • This function is non-deterministic when multiple rows share the minimum ordering value. Results may differ from Spark in that case.

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • The value and ordering must both be fixed-length types (boolean, integral, floating-point, decimal, date, or timestamp). A variable-length or nested type such as string, binary, or struct falls back to Spark.

Mode#

The following incompatibilities cause Mode to fall back to Spark by default. Set spark.comet.expression.Mode.allowIncompatible=true to enable Comet acceleration despite these differences.

  • mode breaks ties non-deterministically in Spark (the result depends on JVM hash-map iteration order); Comet returns the smallest of the tied values instead (https://github.com/apache/datafusion-comet/issues/3970)

Percentile#

The following differences from Spark are always present and do not require any additional configuration:

  • Before Spark 3.5.2, Spark can merge 0.0 and -0.0 into one key and drop the count of one of them (SPARK-45599). Comet keeps every value, so when FLOAT or DOUBLE input holds both zeros the result can differ from Spark’s.

  • From Spark 3.5.2, when FLOAT or DOUBLE input holds both 0.0 and -0.0, Spark ranks first the one its hash table reaches first. Comet replays those tables for each task’s partial buffer and each merge stage, but not Spark’s sort-based fallback once a task holds spark.sql.objectHashAggregate.sortBased.fallbackThreshold groups, nor Comet’s own aggregates emitting groups early or spilling under memory pressure, where a spill inside a merge stage nests that stage’s partials one level deeper than Spark does. In those cases Comet’s zero can differ from Spark’s.

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • An array of percentages is not supported.

  • The percentage argument must be a literal.

  • A frequency argument is not supported.

  • Descending order in WITHIN GROUP (ORDER BY ... DESC) is not supported.

  • Only numeric input types are supported.

PivotFirst#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • Value data type other than Boolean, Byte, Short, Int, Long, Float, Double or Decimal

  • Pivot column of a type other than Boolean, Byte, Short, Int, Long, Float, Double, Decimal, String (UTF8_BINARY collation), Date, Timestamp or TimestampNTZ, which includes the struct that a pivot on more than one column builds

  • Pivot values list is empty

  • Pivot values list contains duplicates, including 0.0 next to -0.0