Date/Time Expressions

• Hour, Minute, Second: Incorrectly apply timezone conversion to TimestampNTZ inputs. TimestampNTZ stores local time without timezone, so no conversion should be applied. These expressions work correctly with Timestamp inputs. #3180

• TruncTimestamp (date_trunc): Produces incorrect results when used with non-UTC timezones. Compatible when timezone is UTC. TimestampNTZ inputs are handled correctly (timezone-independent truncation). #2649

Date and Time Functions

Comet’s native implementation of date and time functions may produce different results than Spark for dates far in the future (approximately beyond year 2100). This is because Comet uses the chrono-tz library for timezone calculations, which has limited support for Daylight Saving Time (DST) rules beyond the IANA time zone database’s explicit transitions.

For dates within a reasonable range (approximately 1970-2100), Comet’s date and time functions are compatible with Spark. For dates beyond this range, functions that involve timezone-aware calculations (such as date_trunc with timezone-aware timestamps) may produce results with incorrect DST offsets.

If you need to process dates far in the future with accurate timezone handling, consider:

• Using timezone-naive types (timestamp_ntz) when timezone conversion is not required

• Falling back to Spark for these specific operations

DateFormatClass

The following incompatibilities cause DateFormatClass to fall back to Spark by default. Set spark.comet.expression.DateFormatClass.allowIncompatible=true to enable Comet acceleration despite these differences.

• Non-UTC timezones may produce different results than Spark

The following cases are not supported by Comet:

• Only the following formats are supported:

  • EEE

  • EEEE

  • HH

  • HH:mm

  • HH:mm:ss

  • MM

  • MMM

  • MMMM

  • dd

  • h:mm a

  • hh:mm a

  • hh:mm:ss a

  • mm

  • ss

  • yy

  • yyyy

  • yyyy-MM-dd

  • yyyy-MM-dd HH:mm:ss

  • yyyy-MM-dd'T'HH:mm:ss

  • yyyy/MM/dd

  • yyyy/MM/dd HH:mm:ss

  • yyyyMM

  • yyyyMMdd

FromUnixTime

The following incompatibilities cause FromUnixTime to fall back to Spark by default. Set spark.comet.expression.FromUnixTime.allowIncompatible=true to enable Comet acceleration despite these differences.

• Only supports the default datetime format pattern yyyy-MM-dd HH:mm:ss. DataFusion’s valid timestamp range differs from Spark (https://github.com/apache/datafusion/issues/16594)

Hour

The following incompatibilities cause Hour to fall back to Spark by default. Set spark.comet.expression.Hour.allowIncompatible=true to enable Comet acceleration despite these differences.

• Incorrectly applies timezone conversion to TimestampNTZ inputs (https://github.com/apache/datafusion-comet/issues/3180)

Minute

The following incompatibilities cause Minute to fall back to Spark by default. Set spark.comet.expression.Minute.allowIncompatible=true to enable Comet acceleration despite these differences.

• Incorrectly applies timezone conversion to TimestampNTZ inputs (https://github.com/apache/datafusion-comet/issues/3180)

Second

The following incompatibilities cause Second to fall back to Spark by default. Set spark.comet.expression.Second.allowIncompatible=true to enable Comet acceleration despite these differences.

• Incorrectly applies timezone conversion to TimestampNTZ inputs (https://github.com/apache/datafusion-comet/issues/3180)

TruncDate

The following incompatibilities cause TruncDate to fall back to Spark by default. Set spark.comet.expression.TruncDate.allowIncompatible=true to enable Comet acceleration despite these differences.

• Non-literal format strings will throw an exception instead of returning NULL

The following cases are not supported by Comet:

• Only the following formats are supported: year, yyyy, yy, quarter, mon, month, mm, week

TruncTimestamp

The following incompatibilities cause TruncTimestamp to fall back to Spark by default. Set spark.comet.expression.TruncTimestamp.allowIncompatible=true to enable Comet acceleration despite these differences.

• Produces incorrect results when used with non-UTC timezones. Compatible when timezone is UTC. (https://github.com/apache/datafusion-comet/issues/2649)

UnixTimestamp

The following cases are not supported by Comet:

• Only TimestampType and DateType inputs are supported. TimestampNTZType is not supported because Comet incorrectly applies timezone conversion to TimestampNTZ values.