Miscellaneous Expressions#

CheckOverflow#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • Only DecimalType is supported

HllSketchEstimate#

The following incompatibilities cause HllSketchEstimate to fall back to Spark by default. Set spark.comet.expression.HllSketchEstimate.allowIncompatible=true to enable Comet acceleration despite these differences.

  • Comet uses a Rust DataSketches port; HLL estimates may differ slightly from Spark.

  • Errors surface as plain Comet execution errors rather than Spark’s SparkRuntimeException with condition HLL_INVALID_INPUT_SKETCH_BUFFER (sqlState 22000), so the message and error class differ even though both engines fail.

HllUnion#

The following incompatibilities cause HllUnion to fall back to Spark by default. Set spark.comet.expression.HllUnion.allowIncompatible=true to enable Comet acceleration despite these differences.

  • Comet uses a Rust DataSketches port; HLL sketch bytes and estimates may differ slightly from Spark.

  • Errors surface as plain Comet execution errors rather than Spark’s SparkRuntimeException with condition HLL_UNION_DIFFERENT_LG_K / HLL_INVALID_INPUT_SKETCH_BUFFER (sqlState 22000), so the message and error class differ even though both engines fail.

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • The allowDifferentLgConfigK argument must be a foldable literal.

Literal#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • Not all data types are supported for literal values

MakeDecimal#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • Only LongType input is supported

ScalarSubquery#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • Not all data types are supported for scalar subquery results

SortOrder#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • A nested sort key with nondefault root null placement must be deterministic: the wrapper can evaluate the key twice, for isnull(key) and its value, which could disagree

StaticInvoke#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • Iceberg’s truncate(width, value) system function on a decimal column. Iceberg’s TruncateDecimal returns a Decimal that can exceed the column’s declared precision, and Spark only turns that into null when the row is materialized. An Arrow Decimal128(precision, scale) array cannot carry that intermediate, so a native kernel would null it during evaluation and change what an enclosing predicate or hash sees. Truncating -99999999999999.9999 in a decimal(18,4) column by a width of 10 is one such value: the result has 19 digits. The other truncate input types, and bucket on decimals, are unaffected.

ToPrettyString#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • Falls back to Spark when the input type cannot be cast to string.

Uuid#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • uuid requires a resolved random seed