Expression Compatibility (Spark 3.4)

Compatibility notes for Comet running on Apache Spark 3.4. Expressions that are not 100% Spark-compatible fall back to Spark by default, except those with a JVM codegen-dispatch path, which stay in Comet’s native pipeline and match Spark exactly. Set spark.comet.expression.EXPRNAME.allowIncompatible=true, where EXPRNAME is the Spark expression class name, to run Comet’s faster native implementation despite its differences from Spark. See the Comet Supported Expressions Guide for more information on this configuration setting.

• Aggregate Expressions
• Array Expressions
• Date/Time Expressions
• Map Expressions
• Math Expressions
• Miscellaneous Expressions
• String Expressions
• Struct Expressions
• URL Expressions
• Cast