conversion_funcs Expression Audits#

Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide Spark Expression Support for current support status.

cast#

  • Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8 modulo Cast.canUpCast refactored to delegate to UpCastRule.canUpCast.

  • Spark 3.5.8 (audited 2026-05-27): baseline. Cast(child, dataType, timeZoneId, evalMode); eval modes are LEGACY, ANSI, TRY. The legacy Cast.canCast matrix and the Cast.canAnsiCast matrix decide acceptance per type pair. Comet routes via CometCast (spark/src/main/scala/org/apache/comet/expressions/CometCast.scala) using a per-source-type support matrix that returns Compatible, Incompatible(reason), or Unsupported(reason); literal children are short-circuited to Compatible() so CometLiteral validates them. The serialized Cast proto carries datatype, evalMode, timezone (default UTC), and isSpark4Plus; allowIncompatible is enforced only by the JVM planner and is not sent to the native cast kernel. The native side (native/spark-expr/src/conversion_funcs/cast.rs) implements explicit per-eval-mode branches for narrowing numeric casts that match Spark’s overflow exceptions, and falls through to DataFusion cast_with_options(safe = !ANSI) for the rest.

  • Spark 4.0.1 (audited 2026-05-27): VariantType added; StringType literals replaced with _: StringType to accommodate collated strings. (TimestampType, ByteType|ShortType|IntegerType) added to canAnsiCast. NullIntolerant -> nullIntolerant: Boolean refactor. New ToPrettyString.BinaryFormatter semantics for Binary -> String are replicated natively via spark_binary_formatter. Numeric-to-numeric matrix unchanged.

  • Spark 4.1.1 (audited 2026-05-27): TimeType added; many TimeType arms in canCast/canAnsiCast. Geospatial GeographyType / GeometryType types added with their own conversion rules. Numeric-to-numeric matrix unchanged.

  • Spark 4.0.5, 4.1.4 and 4.2.1 (release candidates; in 4.0.5-rc3, 4.1.4-rc4 and 4.2.1-rc3): Decimal.fromString and fromStringANSI no longer count integral digits for zero (SPARK-60118), so a zero with a large exponent such as '0e39' casts to 0 instead of NULL (or NUMERIC_OUT_OF_SUPPORTED_RANGE under ANSI). CometCast sets the decimal_zero_any_exponent cast option from SparkFixes.castsZeroWithAnyExponentToDecimal, and parse_string_to_decimal skips the zero limit when it is set. The same fix counts the integral digits in a long. Released Spark counts them in an int, which wraps for an exponent near the int limit, so '1e2147483647', '12e2147483647' and '1.5e2147483647' skip the limit and throw a raw ArithmeticException in every eval mode, and '0e2147483647' casts to 0. On the fixed releases the first three return NULL, or raise NUMERIC_OUT_OF_SUPPORTED_RANGE under ANSI. The native cast returns NULL for them outside ANSI, as the fixed releases do, and raises CAST_INVALID_INPUT under ANSI (#7188). Without decimal_zero_any_exponent it lets '0e2147483647' through as released Spark does. The same releases skip BigDecimal.setScale in Decimal.changePrecision when the value’s magnitude already decides the result (SPARK-60119), so a string far below the target scale’s ulp casts to the rounded value instead of throwing ArithmeticException. '1e-2147483647' to DECIMAL(10,2) gives 0.00, which the native cast already returns, so that case of #6966 applies only to earlier releases. Its negative-scale case (spark.sql.legacy.allowNegativeScaleOfDecimal=true) still differs: '1e-2147483647' to DECIMAL(10,-2) gives 0 on these releases, while the native cast raises CAST_INVALID_INPUT under ANSI and returns NULL otherwise.

  • Known divergences and gaps:

    • CAST(<binary> AS STRING) decodes the bytes with decode_utf8_spark_lossy, replacing invalid UTF-8 with U+FFFD exactly as the JVM’s new String(bytes, UTF_8) does (the same decoder as the native shuffle). It matches Spark’s rendered output, but decoding is not byte-preserving, so it diverges from Spark for operations that read the underlying bytes: round-trips such as CAST(CAST(x AS STRING) AS BINARY), and value identity, since distinct ill-formed byte sequences all decode to U+FFFD and can therefore compare equal in Comet where Spark compares the raw bytes (#4764).

    • Spark 4.0 collated StringType is not explicitly guarded; pattern equality is expected to keep collated-string casts falling back, but there is no test (#4489; umbrella #2190).

    • Spark 4.1 TimeType casts have no explicit Unsupported arm; they fall back implicitly but do not appear in the auto-generated compatibility doc (#4490).

    • CAST(<map> AS <map>) runs natively via cast_map_to_map; CometCast.isSupported recurses into the key and value casts, so the map arm inherits whichever support level the inner casts report.

    • spark.sql.legacy.castComplexTypesToString.enabled=true is Unsupported for any array/map/struct-to-string cast, so those casts fall back to Spark while the flag is on. Comet only implements the default ({}-wrapped, NULL-rendering) formatting. The flag is internal from Spark 4.0 onward and defaults to false.

    • CAST(<float|double> AS DECIMAL) is Compatible: it rounds the shortest decimal string form of the value (matching Spark’s Double.toString + BigDecimal.setScale(HALF_UP) path) rather than the binary value, taking a guarded binary fast path for values provably far from a rounding tie. NaN and infinity are null even in ANSI mode, matching Spark, which swallows the NumberFormatException from Decimal(double) before the ANSI overflow check. The support level carries a note: Double.toString only emits the shortest round-trip form on JDK 19 and later, so on older JDKs Spark’s own result can differ for a value whose shortest form lands exactly on a rounding tie at the target scale.

  • Spark registers the type-name conversion functions (bigint, binary, boolean, date, decimal, double, float, int, smallint, string, timestamp, tinyint) as cast aliases. Each lowers to the same Cast node, so Comet handles it via the cast implementation with the same compatibility profile.

  • Performance (tuned 2026-07-14, PR #4920): narrowing integer casts (spark_cast_int_to_int) map the values buffer in a single pass with Arrow unary/try_unary and carry the null buffer over untouched, replacing an element-by-element Option/Result iterator-collect. Up to 100x faster on narrowing casts. Benchmark: benches/cast_numeric.rs.

  • Performance (tuned 2026-07-15, PR #4940): float/double-to-decimal casts (cast_floating_point_to_decimal128) now convert in a single vectorized unary_opt pass that maps out-of-range values (NaN, infinity, precision overflow) to null, replacing the per-element Decimal128Builder loop. ANSI raises via an O(1) null-count check plus a rare element-wise rescan. 15-36% faster with no regression on any shape. Benchmark: benches/cast_float_to_decimal.rs.

  • Performance (tuned 2026-07-15, PR #4939): integer-to-decimal casts (cast_int_to_decimal128_internal) now convert in a single vectorized unary_opt pass that maps overflowing values to null, replacing the per-element Decimal128Builder loop. ANSI raises via an O(1) null-count check plus a rare element-wise rescan. 28-50% faster with no regression on any shape. Benchmark: benches/cast_int_to_decimal.rs.

  • Performance (tuned 2026-09-30, PR #5130): string-to-timestamp and string-to-timestamp_ntz (timestamp_parser / timestamp_ntz_parser) classify the input shape once with a single byte scan, replacing up to 34 sequential regex is_match calls per row (27 for the NTZ variant). The shapes only accept ASCII digits, so the cast no longer runs a regex at all. 40-86% faster (1.7x-6.9x, median 3.0x) across all 118 benchmark cases, none regressed. Benchmark: benches/cast_string_to_timestamp.rs.

  • Performance (tuned 2026-10-14, PR #6969): casts to timestamp from int, decimal, float, and boolean (cast_int_to_timestamp, cast_decimal_to_timestamp, cast_float_to_timestamp, cast_boolean_to_timestamp) use Arrow unary / try_unary / unary_opt / from_unary instead of a per-row TimestampMicrosecondBuilder. Integer casts are about 9-12x faster and boolean casts about 5-7x. Float casts are 14-35% faster with few nulls and about 4-16x faster when most rows are null, because try_unary and unary_opt skip null slots. Decimal uses one unary_opt pass: about 9% faster with no nulls (under 10%; the 256-bit division dominates), about 19% with sparse nulls, about 2.7x with dense nulls, and about 16x when every row is null. No shape regressed. Benchmarks: benches/cast_int_to_timestamp.rs, benches/cast_non_int_numeric_timestamp.rs.

On this page