Aggregate Expressions#

ApproximatePercentile#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • The percentage argument must be a foldable literal.

  • The accuracy argument must be a foldable literal.

  • Only byte, short, int, long, float, and double input types are supported.

Average#

The following incompatibilities cause Average to fall back to Spark by default. Set spark.comet.expression.Average.allowIncompatible=true to enable Comet acceleration despite these differences.

  • For a DECIMAL input with precision 25 or more, other than DECIMAL(38, 38), the native average rescales the sum within 38 digits before dividing it by the count, so a large sum returns NULL where Spark returns the average. Under ANSI, an average that Spark rejects with NUMERIC_VALUE_OUT_OF_RANGE also returns NULL (#7007).

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • YearMonthIntervalType and DayTimeIntervalType inputs are not supported

CollectSet#

The following differences from Spark are always present and do not require any additional configuration:

  • On Spark 4.2+, collect_set inputs are normalized with Spark’s recursive floating-point normalizer to match SPARK-57298. For array inputs containing floating-point values the normalizer produces ArrayTransform, which Comet executes through the JVM codegen dispatcher instead of fully natively (scalar and struct inputs stay native). When spark.comet.exec.scalaUDF.codegen.enabled=false, those array cases fall back to Spark.

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • collect_set with RESPECT NULLS falls back to Spark, since the native implementation always drops null inputs.

First#

The following differences from Spark are always present and do not require any additional configuration:

  • This function is not deterministic. Results may not match Spark.

HllSketchAgg#

The following incompatibilities cause HllSketchAgg to fall back to Spark by default. Set spark.comet.expression.HllSketchAgg.allowIncompatible=true to enable Comet acceleration despite these differences.

  • Comet uses a Rust DataSketches port; HLL sketch bytes and estimates may differ slightly from Spark.

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • The lgConfigK argument must be a foldable literal.

  • The lgConfigK argument must be in the range [4, 21].

  • Only int, long, string, and binary input types are supported.

HllUnionAgg#

The following incompatibilities cause HllUnionAgg to fall back to Spark by default. Set spark.comet.expression.HllUnionAgg.allowIncompatible=true to enable Comet acceleration despite these differences.

  • Comet uses a Rust DataSketches port; HLL sketch bytes and estimates may differ slightly from Spark.

  • Errors surface as plain Comet execution errors rather than Spark’s SparkRuntimeException with condition HLL_UNION_DIFFERENT_LG_K / HLL_INVALID_INPUT_SKETCH_BUFFER (sqlState 22000), so the message and error class differ even though both engines fail.

  • An input sketch in the updatable HLL_4 form carrying auxiliary-map entries is rejected with an error, where Spark reads it: the bundled Rust decoder reads the compact auxiliary layout in both forms and would otherwise return a silently wrong estimate. Comet only ever writes HLL_8, so this affects sketch columns produced elsewhere.

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • The allowDifferentLgConfigK argument must be a foldable literal.

HyperLogLogPlusPlus#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • Input type must be boolean, integral, floating-point, decimal, date/time, string, or binary

  • DecimalType with precision > 18 is not supported

  • Collated (non-UTF8_BINARY) strings are not supported

Last#

The following differences from Spark are always present and do not require any additional configuration:

  • This function is not deterministic. Results may not match Spark.

ListAgg#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • BinaryType inputs are not supported.

  • WITHIN GROUP (ORDER BY ...) is not supported.

  • Non-literal delimiters are not supported.

  • Non-default string collations are not supported.

Max#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • Input of a type other than boolean, integral, floating-point, decimal, date, timestamp, string or binary, such as timestamp without time zone, intervals, arrays and structs.

  • A string with a non-default collation (Spark 4.0+), which Spark orders by its collation while Comet compares raw bytes.

MaxBy#

The following differences from Spark are always present and do not require any additional configuration:

  • This function is non-deterministic when multiple rows share the maximum ordering value. Results may differ from Spark in that case.

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • The value and ordering must both be fixed-length types (boolean, integral, floating-point, decimal, date, or timestamp). A variable-length or nested type such as string, binary, or struct falls back to Spark.

Min#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • Input of a type other than boolean, integral, floating-point, decimal, date, timestamp, string or binary, such as timestamp without time zone, intervals, arrays and structs.

  • A string with a non-default collation (Spark 4.0+), which Spark orders by its collation while Comet compares raw bytes.

MinBy#

The following differences from Spark are always present and do not require any additional configuration:

  • This function is non-deterministic when multiple rows share the minimum ordering value. Results may differ from Spark in that case.

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • The value and ordering must both be fixed-length types (boolean, integral, floating-point, decimal, date, or timestamp). A variable-length or nested type such as string, binary, or struct falls back to Spark.

Mode#

The following incompatibilities cause Mode to fall back to Spark by default. Set spark.comet.expression.Mode.allowIncompatible=true to enable Comet acceleration despite these differences.

  • mode breaks ties non-deterministically in Spark (the result depends on JVM hash-map iteration order); Comet returns the smallest of the tied values instead (https://github.com/apache/datafusion-comet/issues/3970)

Percentile#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • An array of percentages is not supported.

  • The percentage argument must be a literal.

  • A frequency argument is not supported.

  • Descending order in WITHIN GROUP (ORDER BY ... DESC) is not supported.

  • Only numeric input types are supported.

PivotFirst#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • Value data type other than Boolean, Byte, Short, Int, Long, Float, Double or Decimal

  • Pivot column of a type other than Boolean, Byte, Short, Int, Long, Float, Double, Decimal, String (UTF8_BINARY collation), Date, Timestamp or TimestampNTZ, which includes the struct that a pivot on more than one column builds

  • Pivot values list is empty

  • Pivot values list contains duplicates, including 0.0 next to -0.0