String Expressions#

BitLength#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • BinaryType input is not supported

Concat#

By default, Concat is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set spark.comet.expression.Concat.allowIncompatible=true to opt into Comet’s native implementation instead, which has the following differences from Spark:

  • concat does not support non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/2190)

The following cases have no native implementation and always run in the JVM using Spark’s code-generated implementation (inside the Comet pipeline):

  • CONCAT supports only string input parameters

Contains#

The following cases have no native implementation and always run in the JVM using Spark’s code-generated implementation (inside the Comet pipeline):

  • Non-UTF8_BINARY collated operands are routed through the JVM codegen dispatcher (Spark’s own doGenCode) because native comparison is byte-wise.

EndsWith#

The following cases have no native implementation and always run in the JVM using Spark’s code-generated implementation (inside the Comet pipeline):

  • Non-UTF8_BINARY collated operands are routed through the JVM codegen dispatcher (Spark’s own doGenCode) because native comparison is byte-wise.

GetJsonObject#

By default, GetJsonObject is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set spark.comet.expression.GetJsonObject.allowIncompatible=true to opt into Comet’s native implementation instead, which has the following differences from Spark:

  • Spark allows single-quoted JSON and unescaped control characters which Comet does not support

  • For JSON objects containing duplicate keys, Spark returns the value of the first occurrence while Comet’s native implementation returns the last occurrence (#4947)

InitCap#

By default, InitCap is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set spark.comet.expression.InitCap.allowIncompatible=true to opt into Comet’s native implementation instead, which has the following differences from Spark:

  • Treats hyphen as a word separator (e.g. robert rose-smith produces Robert Rose-Smith instead of Spark’s Robert Rose-smith) (https://github.com/apache/datafusion-comet/issues/1052)

Length#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • BinaryType input is not supported

Levenshtein#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • Non-default collation (non-UTF8_BINARY) is not supported

Like#

The following cases have no native implementation and always run in the JVM using Spark’s code-generated implementation (inside the Comet pipeline):

  • LIKE with a custom escape character (only \ is supported natively)

  • Non-UTF8_BINARY collated operands are routed through the JVM codegen dispatcher (Spark’s own doGenCode) because native comparison is byte-wise.

Lower#

By default, Lower is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set spark.comet.caseConversion.enabled=true to opt into Comet’s native implementation instead, which has the following differences from Spark:

  • Results can vary depending on locale and character set

OctetLength#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • BinaryType input is not supported

RLike#

By default, RLike is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set spark.comet.expression.RLike.allowIncompatible=true to opt into Comet’s native implementation instead, which has the following differences from Spark:

  • Uses Rust regexp engine, which has different behavior to Java regexp engine

RegExpReplace#

By default, RegExpReplace is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set spark.comet.expression.RegExpReplace.allowIncompatible=true to opt into Comet’s native implementation instead, which has the following differences from Spark:

  • Regexp pattern may not be compatible with Spark

Reverse#

By default, Reverse is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set spark.comet.expression.Reverse.allowIncompatible=true to opt into Comet’s native implementation instead, which has the following differences from Spark:

  • native reverse does not support arrays whose element type contains binary, struct, or map

  • reverse does not support non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/2190)

StartsWith#

The following cases have no native implementation and always run in the JVM using Spark’s code-generated implementation (inside the Comet pipeline):

  • Non-UTF8_BINARY collated operands are routed through the JVM codegen dispatcher (Spark’s own doGenCode) because native comparison is byte-wise.

StringLPad#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • Scalar values are not supported for the str argument.

  • Only scalar values are supported for the pad argument.

StringRPad#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • Scalar values are not supported for the str argument.

  • Only scalar values are supported for the pad argument.

StringRepeat#

The following differences from Spark are always present and do not require any additional configuration:

  • A negative argument for the number of times to repeat throws an exception instead of returning an empty string as Spark does

StringReplace#

By default, StringReplace is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set spark.comet.expression.StringReplace.allowIncompatible=true to opt into Comet’s native implementation instead, which has the following differences from Spark:

  • Produces different results from Spark when the search string is empty

StringSplit#

By default, StringSplit is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set spark.comet.expression.StringSplit.allowIncompatible=true to opt into Comet’s native implementation instead, which has the following differences from Spark:

  • Regex engine differences between Java and Rust

StringTranslate#

The following incompatibilities cause StringTranslate to fall back to Spark by default. Set spark.comet.expression.StringTranslate.allowIncompatible=true to enable Comet acceleration despite these differences.

  • DataFusion’s translate iterates over Unicode graphemes (Spark uses code points) and substitutes U+0000 instead of treating it as a deletion sentinel

Upper#

By default, Upper is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set spark.comet.caseConversion.enabled=true to opt into Comet’s native implementation instead, which has the following differences from Spark:

  • Results can vary depending on locale and character set