Spark Version Compatibility

This page documents known issues and limitations specific to each supported Apache Spark version.

For general compatibility information that applies across all Spark versions, see the other pages in this compatibility guide.

Spark 3.4

Spark 3.4.3 is supported with Java 11/17 and Scala 2.12/2.13.

Spark 3.5

Spark 3.5.8 is supported with Java 11/17 and Scala 2.12/2.13.

Spark 4.0

Spark 4.0.2 is supported with Java 17 and Scala 2.13.

Known Limitations

• Collation support (#1947, #4051): Spark 4.0 introduced collation support. Non-default collated strings are not yet supported by Comet and will fall back to Spark.

Spark 4.1 (Experimental)

Spark 4.1.1 is provided as experimental support with Java 17 and Scala 2.13.

Warning

Spark 4.1 support is experimental and intended for development and testing only. It should not be used in production.

Spark 4.2 (Experimental)

Spark 4.2.0-preview4 is provided as experimental support with Java 17 and Scala 2.13.

Warning

Spark 4.2 support is experimental and targets a preview release of Spark. It is intended for early evaluation only and should not be used in production.