Aggregate Expressions#

ApproximatePercentile#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • The percentage argument must be a foldable literal.

  • The accuracy argument must be a foldable literal.

  • Only byte, short, int, long, float, and double input types are supported.

Average#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • YearMonthIntervalType and DayTimeIntervalType inputs are not supported

CollectSet#

The following incompatibilities cause CollectSet to fall back to Spark by default. Set spark.comet.expression.CollectSet.allowIncompatible=true to enable Comet acceleration despite these differences.

  • Before Spark 4.2, Comet deduplicates NaN values (treats NaN == NaN) while Spark treats each NaN as a distinct value. Comet treats -0.0 and 0.0 as distinct while Spark treats them as equal. When spark.comet.exec.strictFloatingPoint=true, collect_set on floating-point types falls back to Spark on those versions unless spark.comet.expression.CollectSet.allowIncompatible=true is set.

First#

The following differences from Spark are always present and do not require any additional configuration:

  • This function is not deterministic. Results may not match Spark.

HllSketchAgg#

The following incompatibilities cause HllSketchAgg to fall back to Spark by default. Set spark.comet.expression.HllSketchAgg.allowIncompatible=true to enable Comet acceleration despite these differences.

  • Comet uses a Rust DataSketches port; HLL sketch bytes and estimates may differ slightly from Spark.

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • The lgConfigK argument must be a foldable literal.

  • The lgConfigK argument must be in the range [4, 21].

  • Only int, long, string, and binary input types are supported.

HllUnionAgg#

The following incompatibilities cause HllUnionAgg to fall back to Spark by default. Set spark.comet.expression.HllUnionAgg.allowIncompatible=true to enable Comet acceleration despite these differences.

  • Comet uses a Rust DataSketches port; HLL sketch bytes and estimates may differ slightly from Spark.

  • Errors surface as plain Comet execution errors rather than Spark’s SparkRuntimeException with condition HLL_UNION_DIFFERENT_LG_K / HLL_INVALID_INPUT_SKETCH_BUFFER (sqlState 22000), so the message and error class differ even though both engines fail.

  • An input sketch in the updatable HLL_4 form carrying auxiliary-map entries is rejected with an error, where Spark reads it: the bundled Rust decoder reads the compact auxiliary layout in both forms and would otherwise return a silently wrong estimate. Comet only ever writes HLL_8, so this affects sketch columns produced elsewhere.

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • The allowDifferentLgConfigK argument must be a foldable literal.

HyperLogLogPlusPlus#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • Input type must be boolean, integral, floating-point, decimal, date/time, string, or binary

  • DecimalType with precision > 18 is not supported

  • Collated (non-UTF8_BINARY) strings are not supported

Last#

The following differences from Spark are always present and do not require any additional configuration:

  • This function is not deterministic. Results may not match Spark.

ListAgg#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • BinaryType inputs are not supported.

  • WITHIN GROUP (ORDER BY ...) is not supported.

  • Non-literal delimiters are not supported.

  • Non-default string collations are not supported.

Max#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • Input of a type other than boolean, integral, floating-point, decimal, date, timestamp, string or binary, such as timestamp without time zone, intervals, arrays and structs.

  • A string with a non-default collation (Spark 4.0+), which Spark orders by its collation while Comet compares raw bytes.

MaxBy#

The following differences from Spark are always present and do not require any additional configuration:

  • This function is non-deterministic when multiple rows share the maximum ordering value. Results may differ from Spark in that case.

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • The value and ordering must both be fixed-length types (boolean, integral, floating-point, decimal, date, or timestamp). A variable-length or nested type such as string, binary, or struct falls back to Spark.

Min#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • Input of a type other than boolean, integral, floating-point, decimal, date, timestamp, string or binary, such as timestamp without time zone, intervals, arrays and structs.

  • A string with a non-default collation (Spark 4.0+), which Spark orders by its collation while Comet compares raw bytes.

MinBy#

The following differences from Spark are always present and do not require any additional configuration:

  • This function is non-deterministic when multiple rows share the minimum ordering value. Results may differ from Spark in that case.

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • The value and ordering must both be fixed-length types (boolean, integral, floating-point, decimal, date, or timestamp). A variable-length or nested type such as string, binary, or struct falls back to Spark.

Mode#

The following incompatibilities cause Mode to fall back to Spark by default. Set spark.comet.expression.Mode.allowIncompatible=true to enable Comet acceleration despite these differences.

  • mode breaks ties non-deterministically in Spark (the result depends on JVM hash-map iteration order); Comet returns the smallest of the tied values instead (https://github.com/apache/datafusion-comet/issues/3970)

Percentile#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • An array of percentages is not supported.

  • The percentage argument must be a literal.

  • A frequency argument is not supported.

  • Descending order in WITHIN GROUP (ORDER BY ... DESC) is not supported.

  • Only numeric input types are supported.