Miscellaneous Expressions#

CheckOverflow#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • Only DecimalType is supported

HllSketchEstimate#

The following incompatibilities cause HllSketchEstimate to fall back to Spark by default. Set spark.comet.expression.HllSketchEstimate.allowIncompatible=true to enable Comet acceleration despite these differences.

  • Comet uses a Rust DataSketches port; HLL estimates may differ slightly from Spark.

  • Errors surface as plain Comet execution errors rather than Spark’s SparkRuntimeException with condition HLL_INVALID_INPUT_SKETCH_BUFFER (sqlState 22000), so the message and error class differ even though both engines fail.

  • An input sketch in the updatable HLL_4 form carrying auxiliary-map entries is rejected with an error, where Spark reads it: the bundled Rust decoder reads the compact auxiliary layout in both forms and would otherwise return a silently wrong estimate. Comet only ever writes HLL_8, so this affects sketch columns produced elsewhere.

HllUnion#

The following incompatibilities cause HllUnion to fall back to Spark by default. Set spark.comet.expression.HllUnion.allowIncompatible=true to enable Comet acceleration despite these differences.

  • Comet uses a Rust DataSketches port; HLL sketch bytes and estimates may differ slightly from Spark.

  • Errors surface as plain Comet execution errors rather than Spark’s SparkRuntimeException with condition HLL_UNION_DIFFERENT_LG_K / HLL_INVALID_INPUT_SKETCH_BUFFER (sqlState 22000), so the message and error class differ even though both engines fail.

  • An input sketch in the updatable HLL_4 form carrying auxiliary-map entries is rejected with an error, where Spark reads it: the bundled Rust decoder reads the compact auxiliary layout in both forms and would otherwise return a silently wrong estimate. Comet only ever writes HLL_8, so this affects sketch columns produced elsewhere.

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • The allowDifferentLgConfigK argument must be a foldable literal.

Literal#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • Not all data types are supported for literal values

MakeDecimal#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • Only LongType input is supported

ScalarSubquery#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • Not all data types are supported for scalar subquery results

SortOrder#

The following incompatibilities cause SortOrder to fall back to Spark by default. Set spark.comet.expression.SortOrder.allowIncompatible=true to enable Comet acceleration despite these differences.

  • Sorting on floating-point values nested in arrays, structs, or maps is not 100% compatible with Spark when spark.comet.exec.strictFloatingPoint=true

StaticInvoke#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • Iceberg’s truncate(width, value) system function on a decimal column. Iceberg’s TruncateDecimal returns a Decimal that can exceed the column’s declared precision, and Spark only turns that into null when the row is materialized. An Arrow Decimal128(precision, scale) array cannot carry that intermediate, so a native kernel would null it during evaluation and change what an enclosing predicate or hash sees. Truncating -99999999999999.9999 in a decimal(18,4) column by a width of 10 is one such value: the result has 19 digits. The other truncate input types, and bucket on decimals, are unaffected.

ToPrettyString#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • Falls back to Spark when the input type cannot be cast to string.

Uuid#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • uuid requires a resolved random seed