Spark Expression Support#

This page is the complete reference for how Apache Comet handles each Spark built-in expression. Comet accelerates expressions either with a native (Rust) implementation or by dispatching to a Spark-compatible codegen path. When an expression is not supported, Comet transparently falls back to Spark for that part of the plan; results are unaffected.

Expressions marked ✅ Supported are enabled by default and produce Spark-compatible results.

Some ✅ Supported expressions have specific incompatible cases that are not run by default. Those cases must be opted into per expression with spark.comet.expression.EXPRNAME.allowIncompatible=true (where EXPRNAME is the Spark expression class name, for example Cast). There is no global opt-in. By default such a case either falls back to Spark (for example the mode aggregate) or, when the expression has a Spark-compatible codegen-dispatch implementation, runs through that instead (for example cast and the regex and JSON families). See Native and codegen-dispatch implementations for how Comet chooses.

Most expressions can also be disabled with spark.comet.expression.EXPRNAME.enabled=false, where EXPRNAME is the Spark expression class name (for example Length or StartsWith). See the Comet Configuration Guide for the full list.

Status legend#

Status	Meaning
✅ Supported	Comet produces Spark-compatible results by default. Some inputs or forms may fall back to Spark, and any incompatible behavior is opt-in (off by default).
🔜 Planned	Intended; tracked by an open issue or pull request.

Implementation legend#

The Implementation column records how Comet executes each expression when it is not falling back to Spark:

Implementation	Meaning
Native	Comet’s Rust engine evaluates the expression end to end.
Codegen dispatch	Spark’s own generated JVM code is evaluated inside the Comet pipeline. Used when a byte-exact match to Spark matters more than the native speedup, or when no native path exists.
Hybrid	Both paths exist. Comet picks between them based on input, and the user can override with `spark.comet.expression.EXPRNAME.allowIncompatible=true` — see Native and codegen-dispatch implementations.
—	No direct wire-up: the row is either planned, is rewritten to another expression before Comet sees it (for example `to_date` → `Cast`), or is handled at the operator level rather than the expression level (for example `explode`, window functions).

The Implementation column is auto-generated from the serde definitions in QueryPlanSerde; do not edit it by hand.

Not currently planned#

Comet focuses acceleration on mainstream relational, string, datetime, math, and collection expressions. The following function families are not currently planned for native acceleration (they are not on the current roadmap): specialized functionality with narrow real-world analytics use and high implementation cost. They fall back to Spark and may be reconsidered based on demand:

Probabilistic sketches and approximate top-k (kll_sketch_*, theta_*, count_min_sketch, bitmap_*, approx_top_k*): specialized data structures with exact-correctness traps.
Geospatial (st_*): brand-new Spark 4.1 functionality, specialized.
Avro / Protobuf codecs (from_avro, to_avro, from_protobuf, to_protobuf, schema_of_avro): format conversion belongs at the IO layer, not expression evaluation.
JVM reflection (java_method, reflect, try_reflect): niche, and they invoke arbitrary JVM methods (a security concern).
UTF-8 validation (is_valid_utf8, make_valid_utf8, validate_utf8, try_validate_utf8): niche Spark 4.x string-validation helpers.
Miscellaneous niche (histogram_numeric, version, sentences, quote): low-value or specialized functions with little benefit from native acceleration.

The file-metadata functions input_file_name, input_file_block_start, and input_file_block_length depend on scan-internal per-row file information rather than the expression layer; their support status is covered in the Parquet page.

Note that median and mode are supported: they are mainstream exact aggregates. mode runs natively only with spark.comet.expression.Mode.allowIncompatible=true. approx_count_distinct is supported because Comet ports Spark’s HyperLogLogPlusPlus exactly, so its result is bit-identical to Spark.

The Apache DataSketches HLL functions (hll_sketch_agg, hll_union_agg, hll_sketch_estimate, hll_union) are the one sketch family Comet does accelerate, on Spark 4.0+. The sketches are mutually readable with Spark’s, but the point estimate can differ slightly after a merge, so the native path is off by default and opt-in per expression via allowIncompatible — see the rows below.

The tables below list every Spark built-in expression with its current status.

agg_funcs#

Function	Status	Implementation	Notes
`any`	✅	—
`any_value`	✅	—
`approx_count_distinct`	✅	Native
`approx_percentile`	✅	Native	Byte, short, int, long, float, and double input; other input types fall back to Spark
`array_agg`	✅	Native	Alias for `collect_list`
`avg`	✅	Native	Interval types fall back. For `DECIMAL` input with precision 28 or more, fallback depends on the aggregate operator; see Hash Aggregation
`bit_and`	✅	Native
`bit_or`	✅	Native
`bit_xor`	✅	Native
`bool_and`	✅	—
`bool_or`	✅	—
`collect_list`	✅	Native
`collect_set`	✅	Native
`corr`	✅	Native
`count`	✅	Native
`count_if`	✅	—
`covar_pop`	✅	Native
`covar_samp`	✅	Native
`every`	✅	—
`first`	✅	Native
`first_value`	✅	Native
`grouping`	✅	—	Grouping indicator for ROLLUP/CUBE/GROUPING SETS
`grouping_id`	✅	—	Grouping indicator for ROLLUP/CUBE/GROUPING SETS
`hll_sketch_agg`	✅	Native	Spark 4.0+ only; falls back by default, the native path is opt-in via allowIncompatible (details)
`hll_union_agg`	✅	Native	Spark 4.0+ only; falls back by default, the native path is opt-in via allowIncompatible (details)
`kurtosis`	🔜	—	Not yet implemented natively
`last`	✅	Native
`last_value`	✅	Native
`listagg`	✅	Native	Spark 4.0+. `StringType` input with a literal delimiter; `WITHIN GROUP (ORDER BY ...)` and `BinaryType` inputs fall back to Spark. Without `WITHIN GROUP`, concatenation order is the (non-deterministic) group arrival order, matching Spark.
`max`	✅	Native
`max_by`	✅	Native	Value and ordering must be fixed-length types
`mean`	✅	Native	Interval types fall back. For `DECIMAL` input with precision 28 or more, fallback depends on the aggregate operator; see Hash Aggregation
`median`	✅	—	Rewrites to `percentile(col, 0.5)` and runs natively for supported percentile inputs
`min`	✅	Native
`min_by`	✅	Native	Value and ordering must be fixed-length types
`mode`	✅	—	`mode(col)` only; Spark breaks ties non-deterministically, so Comet returns the smallest tied value and falls back by default, opt-in via allowIncompatible
`percentile`	✅	Native	Single literal percentage on numeric input runs natively; array of percentages and a frequency argument fall back to Spark
`percentile_approx`	✅	Native	Alias of `approx_percentile`; same restrictions apply
`percentile_cont`	✅	—	Spark 4.0+ `WITHIN GROUP (ORDER BY ...)`; ascending only runs natively, `DESC` falls back to Spark
`percentile_disc`	🔜	—	Percentile aggregate
`pivot_first`	✅	—	Internal aggregate for the optimized `PIVOT` fast path. Value type must be in `PivotFirst.supportsDataType` (Boolean, Byte, Short, Int, Long, Float, Double, Decimal); other types keep Spark’s standard filtered-aggregate path. Binary, array and struct pivot columns (including pivots on more than one column), non-default collations and pivot lists with duplicate values fall back to Spark
`regr_avgx`	✅	—	Native: Spark rewrites to `Average` (tests in #4551)
`regr_avgy`	✅	—	Native: Spark rewrites to `Average` (tests in #4551)
`regr_count`	✅	—	Native: Spark rewrites to `Count` (tests in #4551)
`regr_intercept`	✅	Native
`regr_r2`	✅	Native
`regr_slope`	✅	Native
`regr_sxx`	✅	—
`regr_sxy`	✅	Native
`regr_syy`	✅	—
`skewness`	🔜	—	Not yet implemented natively
`some`	✅	—
`std`	✅	Native
`stddev`	✅	Native
`stddev_pop`	✅	Native
`stddev_samp`	✅	Native
`string_agg`	✅	Native	Alias of `listagg`; same restrictions apply.
`sum`	✅	Native
`try_avg`	✅	—	Interval types fall back. For `DECIMAL` input with precision 28 or more, fallback depends on the aggregate operator; see Hash Aggregation
`try_sum`	✅	—
`var_pop`	✅	Native
`var_samp`	✅	Native
`variance`	✅	Native

array_funcs#

Function	Status	Implementation	Notes
`array`	✅	Native
`array_append`	✅	Native
`array_compact`	✅	—
`array_contains`	✅	Hybrid	Flat float/double element arrays use a native kernel with Spark’s equality; nested float elements, non-default string collations, and a value that is not a literal or column over a nullable array, route through the JVM codegen dispatcher
`array_distinct`	✅	Native	Floating-point elements fall back on Spark versions before 4.0.5, 4.1.4, and 4.2; signed-zero results may differ with native opt-in (details). Non-default string collations fall back
`array_except`	✅	Hybrid	Routes through the JVM codegen dispatcher by default; the incompatible native path is opt-in via allowIncompatible (details)
`array_insert`	✅	Native
`array_intersect`	✅	Hybrid	Routes through the JVM codegen dispatcher by default; the incompatible native path is opt-in via allowIncompatible (details)
`array_join`	✅	Hybrid	Native for literal or column delimiter and null replacement; other cases and non-UTF8_BINARY collations use the JVM codegen dispatcher (details)
`array_max`	✅	Hybrid	Native Spark-compatible floating-point and nested ordering; non-default string collations use the JVM codegen dispatcher (details)
`array_min`	✅	Hybrid	Native Spark-compatible floating-point and nested ordering; non-default string collations use the JVM codegen dispatcher (details)
`array_position`	✅	Native	Binary/struct/map/null elements fall back
`array_prepend`	✅	—
`array_remove`	✅	Native
`array_repeat`	✅	Native
`array_union`	✅	Native	Floating-point elements fall back on Spark versions before 4.0.5, 4.1.4, and 4.2; signed-zero results may differ with native opt-in (details). Non-default string collations fall back
`arrays_overlap`	✅	Hybrid	Non-default string collations use the JVM codegen dispatcher
`arrays_zip`	✅	Native
`element_at`	✅	Native
`flatten`	✅	Native	Binary/struct/map elements fall back
`get`	✅	—
`sequence`	✅	Hybrid	Integral types run natively; date/timestamp sequences use codegen dispatch
`shuffle`	✅	Native	Binary/struct/map elements fall back
`slice`	✅	Native	Native (#4149)
`sort_array`	✅	Hybrid	Struct, nested-array, floating-point, and null elements run natively; other element types (for example intervals) route through the JVM codegen dispatcher

bitwise_funcs#

Function	Status	Implementation	Notes
`&`	✅	Native
`<<`	✅	Native
`>>`	✅	Native
`>>>`	✅	Native	Operator alias for `shiftrightunsigned` (Spark 4.0+)
`^`	✅	Native
`bit_count`	✅	Native
`bit_get`	✅	Native
`getbit`	✅	Native
`shiftright`	✅	Native
`shiftrightunsigned`	✅	Native
`\|`	✅	Native
`~`	✅	Native

collection_funcs#

Function	Status	Implementation	Notes
`array_size`	✅	—
`cardinality`	✅	Native
`concat`	✅	Hybrid	Binary/array children and non-UTF8_BINARY collations route through the JVM codegen dispatcher
`reverse`	✅	Hybrid	Binary input (Spark 4.2), arrays with binary, struct, or map elements, and collated strings, route through the JVM codegen dispatcher (details)
`size`	✅	Native

conditional_funcs#

Function	Status	Implementation	Notes
`coalesce`	✅	Native
`if`	✅	Native
`ifnull`	✅	—
`nanvl`	✅	Codegen dispatch
`nullif`	✅	—
`nullifzero`	✅	—	Lowers to `if`/`=` (Spark 4.0+)
`nvl`	✅	—
`nvl2`	✅	—
`when`	✅	Native
`zeroifnull`	✅	—	Lowers to `coalesce` (Spark 4.0+)

conversion_funcs#

The type-name conversion functions (bigint, binary, boolean, date, decimal, double, float, int, smallint, string, timestamp, tinyint) are SQL aliases for CAST(... AS <type>) and share the support and caveats of cast.

Function	Status	Implementation	Notes
`cast`	✅	Hybrid	Casts without a native path (for example boolean to decimal) route through the JVM codegen dispatcher (details)

csv_funcs#

Function	Status	Implementation	Notes
`from_csv`	✅	Codegen dispatch
`schema_of_csv`	✅	Codegen dispatch
`to_csv`	✅	Hybrid	Codegen dispatch by default; the native path is opt-in via allowIncompatible

datetime_funcs#

Function	Status	Implementation	Notes
`add_months`	✅	Codegen dispatch
`convert_timezone`	✅	Hybrid	Routes through the JVM codegen dispatcher by default (handles all timezone forms); the native path is opt-in via allowIncompatible (details)
`curdate`	✅	—	Constant-folded to a literal (alias of `current_date`)
`current_date`	✅	—	Constant-folded to a literal before Comet sees the plan
`current_time`	✅	—	Spark 4.1+; requires `spark.sql.timeType.enabled=true`, which Spark leaves off by default. Constant-folded to a literal before Comet sees the plan
`current_timestamp`	✅	—	Constant-folded to a literal before Comet sees the plan
`current_timezone`	✅	—
`date_add`	✅	Native	The 2-argument form is native; the `date_add(UNIT, n, ts)` form (Spark 3.5+) parses to `timestampadd` and runs through codegen dispatch
`date_diff`	✅	Native	The 2-argument form is native; the `date_diff(UNIT, start, end)` form (Spark 3.5+) parses to `timestampdiff` and runs through codegen dispatch
`date_format`	✅	Hybrid
`date_from_unix_date`	✅	Native
`date_part`	✅	—
`date_sub`	✅	Native
`date_trunc`	✅	Hybrid
`dateadd`	✅	Native	The 2-argument form is native; the `dateadd(UNIT, n, ts)` form parses to `timestampadd` and runs through codegen dispatch
`datediff`	✅	Native	The 2-argument form is native; the `datediff(UNIT, start, end)` form parses to `timestampdiff` and runs through codegen dispatch
`datepart`	✅	—
`day`	✅	Native
`dayname`	✅	—	Abbreviated day name (Spark 4.0+)
`dayofmonth`	✅	Native
`dayofweek`	✅	Native
`dayofyear`	✅	Native
`extract`	✅	—	`SECOND FROM TIME` is native for Spark 4.1+ with `Decimal(8,6)` output
`from_unixtime`	✅	Hybrid
`from_utc_timestamp`	✅	Hybrid	Routes through the JVM codegen dispatcher by default (handles all timezone forms); the native path is opt-in via allowIncompatible (details)
`hour`	✅	—
`last_day`	✅	Native
`localtimestamp`	✅	—
`make_date`	✅	Native	Computed arguments preserve year → month → day NULL checks. Expressions fall back when shared subexpressions or a parent could change which arguments Spark evaluates. IN probes stay native for literal/column candidates and array/struct constructors of those values; other computed candidates require Spark evaluation. Operator-level safeguards and their opt-out are described in make_date evaluation.
`make_dt_interval`	✅	Codegen dispatch
`make_interval`	✅	Hybrid	Routes through the JVM codegen dispatcher by default; intervals outside Arrow’s nanosecond range are tracked by #5279; the native path is opt-in via allowIncompatible (details)
`make_time`	✅	—	Spark 4.1+; requires `spark.sql.timeType.enabled=true`, which Spark leaves off by default. Runs natively; remaining TIME type work is tracked by #4288
`make_timestamp`	✅	—
`make_timestamp_ltz`	✅	—	The 2-argument `(date, time)` form (Spark 4.1+) runs through codegen dispatch
`make_timestamp_ntz`	✅	—	The 2-argument `(date, time)` form (Spark 4.1+) runs through codegen dispatch
`make_ym_interval`	✅	Codegen dispatch
`minute`	✅	—
`month`	✅	Native
`monthname`	✅	—	Abbreviated month name (Spark 4.0+)
`months_between`	✅	Codegen dispatch
`next_day`	✅	Hybrid	Non-UTF8_BINARY collated `dayOfWeek` routes through the JVM codegen dispatcher; other input runs natively
`now`	✅	—	Constant-folded to a literal (alias of `current_timestamp`)
`quarter`	✅	Native
`second`	✅	—
`session_window`	🔜	—	Batch session-window grouping falls back (`UpdatingSessionsExec` is not yet native); tracked by #4785
`time_diff`	✅	—	Spark 4.1+; requires `spark.sql.timeType.enabled=true`, which Spark leaves off by default
`time_trunc`	✅	—	Spark 4.1+; requires `spark.sql.timeType.enabled=true`, which Spark leaves off by default
`timediff`	✅	—	Spark 4.0+ grammar alias that parses to `timestampdiff`; runs through codegen dispatch
`timestamp_micros`	✅	Codegen dispatch
`timestamp_millis`	✅	Codegen dispatch
`timestamp_seconds`	✅	Hybrid	Integer, long, float and double inputs run natively; decimal, byte and short inputs route through the JVM codegen dispatcher
`timestampadd`	✅	—	Reached through the grammar rather than the function registry; runs through codegen dispatch
`timestampdiff`	✅	—	Reached through the grammar rather than the function registry; runs through codegen dispatch
`to_date`	✅	—	Rewrites to `Cast` (or `Cast(GetTimestamp)` with a format) before Comet sees the plan
`to_time`	✅	—	Spark 4.1+; requires `spark.sql.timeType.enabled=true`, which Spark leaves off by default. The one-argument form runs natively, and the form with a format argument runs through codegen dispatch
`to_timestamp`	✅	—	Rewrites to `Cast` (or `GetTimestamp` with a format) before Comet sees the plan
`to_timestamp_ltz`	✅	—	Rewrites to `to_timestamp` (`TimestampType`)
`to_timestamp_ntz`	✅	—	Rewrites to `to_timestamp` (`TimestampNTZType`)
`to_unix_timestamp`	✅	Codegen dispatch
`to_utc_timestamp`	✅	Hybrid	Routes through the JVM codegen dispatcher by default (handles all timezone forms); the native path is opt-in via allowIncompatible (details)
`trunc`	✅	Hybrid
`try_make_interval`	✅	—	Rewrites to `MakeInterval`; same support as `make_interval` (Spark 4.0+)
`try_make_timestamp`	✅	—
`try_make_timestamp_ltz`	✅	—	Same support as `try_make_timestamp` (Spark 4.0+)
`try_make_timestamp_ntz`	✅	—	Same support as `try_make_timestamp` (Spark 4.0+)
`try_to_date`	✅	—	Rewrites to `Cast`/`GetTimestamp` before Comet sees the plan; same support as `to_date`
`try_to_time`	✅	—	Same support as `to_time`
`try_to_timestamp`	✅	—	Rewrites to `Cast`/`GetTimestamp` before Comet sees the plan; same support as `to_timestamp`
`unix_date`	✅	Native
`unix_micros`	✅	Codegen dispatch
`unix_millis`	✅	Codegen dispatch
`unix_seconds`	✅	Codegen dispatch
`unix_timestamp`	✅	Hybrid	String parsing uses Spark’s codegen and honors the time parser policy, ANSI mode, and session time zone. Date and timestamp inputs ignore the format and use native execution.
`weekday`	✅	Native
`weekofyear`	✅	Native
`window`	✅	—	Batch tumbling and sliding time-window grouping runs natively
`window_time`	✅	—	Batch time-window grouping runs natively
`year`	✅	Native

generator_funcs#

explode, explode_outer, posexplode, and posexplode_outer are supported via CometExplodeExec (operator-level, not expression-level) for array input; map input falls back to Spark (#2837). Enabled by default via spark.comet.exec.explode.enabled.

Function	Status	Implementation	Notes
`explode`	✅	—	via `CometExplodeExec`
`explode_outer`	✅	—	via `CometExplodeExec` (audit)
`inline`	🔜	—	Operator-level generator (like `explode`)
`inline_outer`	🔜	—	Operator-level generator (like `explode`)
`posexplode`	✅	—	via `CometExplodeExec`
`posexplode_outer`	✅	—	via `CometExplodeExec` (audit)
`stack`	🔜	—	Operator-level generator

hash_funcs#

Function	Status	Implementation	Notes
`crc32`	✅	Native
`hash`	✅	Native	Includes decimals with precision 19–38 and nested decimals
`md5`	✅	Native
`sha`	✅	Native
`sha1`	✅	Native
`sha2`	✅	Native
`xxhash64`	✅	Native	Includes decimals with precision 19–38 and nested decimals

json_funcs#

Function	Status	Implementation	Notes
`from_json`	✅	Hybrid	Routes through the JVM codegen dispatcher by default; the native path (supported schemas only) is opt-in via allowIncompatible (audit)
`get_json_object`	✅	Hybrid	Routes through the JVM codegen dispatcher by default; the native path is opt-in via allowIncompatible (audit)
`json_array_length`	✅	Hybrid	Routes through the JVM codegen dispatcher by default; the native path, which differs for single-quoted JSON, unescaped control characters, and trailing content, is opt-in via allowIncompatible (audit)
`json_object_keys`	✅	Codegen dispatch
`json_tuple`	🔜	—	#3160
`schema_of_json`	✅	Codegen dispatch
`to_json`	✅	Hybrid	Routes through the JVM codegen dispatcher by default, including options and map/array inputs; the native path is opt-in via allowIncompatible (audit)

lambda_funcs#

Function	Status	Implementation	Notes
`aggregate`	✅	Codegen dispatch
`array_sort`	✅	Native	The default comparator runs natively; custom comparators and arrays with non-default string collations use the JVM codegen dispatcher
`exists`	✅	Codegen dispatch
`filter`	✅	Native	General lambda routed through the JVM codegen dispatcher; the `array_compact` form runs natively
`forall`	✅	Codegen dispatch
`map_filter`	✅	Codegen dispatch
`map_zip_with`	✅	Codegen dispatch
`reduce`	✅	Codegen dispatch
`transform`	✅	Codegen dispatch
`transform_keys`	✅	Codegen dispatch
`transform_values`	✅	Codegen dispatch
`zip_with`	✅	Codegen dispatch

map_funcs#

Function	Status	Implementation	Notes
`element_at`	✅	Native
`map`	✅	Codegen dispatch	Routed through the JVM codegen dispatcher
`map_concat`	✅	Codegen dispatch
`map_contains_key`	✅	—
`map_entries`	✅	Native
`map_from_arrays`	✅	Hybrid	`FLOAT` or `DOUBLE` keys and `spark.sql.mapKeyDedupPolicy=LAST_WIN` route through the JVM codegen dispatcher (details)
`map_from_entries`	✅	Hybrid	BinaryType keys/values, `FLOAT` or `DOUBLE` keys and `spark.sql.mapKeyDedupPolicy=LAST_WIN` route through the JVM codegen dispatcher (details)
`map_keys`	✅	Native
`map_values`	✅	Native
`str_to_map`	✅	Hybrid
`try_element_at`	✅	—	Lowers to `element_at`

math_funcs#

Function	Status	Implementation	Notes
`%`	✅	Native
`*`	✅	Native	YearMonth and DayTime interval multiplication routes through the JVM codegen dispatcher; Calendar interval multiplication falls back
`+`	✅	Native
`-`	✅	Native
`/`	✅	Native	DayTime interval division routes through the JVM codegen dispatcher; YearMonth and Calendar interval division fall back
`abs`	✅	Hybrid	Interval types route through the JVM codegen dispatcher; numeric types run natively
`acos`	✅	Native
`acosh`	✅	Native
`asin`	✅	Native
`asinh`	✅	Native
`atan`	✅	Native
`atan2`	✅	Native
`atanh`	✅	Native
`bin`	✅	Native
`bround`	✅	Codegen dispatch
`cbrt`	✅	Native
`ceil`	✅	—	Two-arg form falls back
`ceiling`	✅	—
`conv`	✅	Codegen dispatch
`cos`	✅	Native
`cosh`	✅	Native
`cot`	✅	Native
`csc`	✅	Native
`degrees`	✅	Native
`div`	✅	Native
`e`	✅	—	Folds to a literal (like `pi`)
`exp`	✅	Native
`expm1`	✅	Native
`factorial`	✅	Native
`floor`	✅	—	Two-arg form falls back
`greatest`	✅	Hybrid	Non-UTF8_BINARY collated input routes through the JVM codegen dispatcher; other input runs natively
`hex`	✅	Native
`hypot`	✅	Codegen dispatch
`least`	✅	Hybrid	Non-UTF8_BINARY collated input routes through the JVM codegen dispatcher; other input runs natively
`ln`	✅	Native
`log`	✅	Native
`log10`	✅	Native
`log1p`	✅	Codegen dispatch
`log2`	✅	Native
`mod`	✅	Native
`negative`	✅	Native
`pi`	✅	Native
`pmod`	✅	Codegen dispatch
`positive`	✅	Codegen dispatch
`pow`	✅	Native
`power`	✅	Native
`radians`	✅	Native
`rand`	✅	Native
`randn`	✅	Native
`random`	✅	Native	Alias for `rand`; seed must be a literal
`randstr`	✅	Native	Random string (Spark 4.0+); length and seed must be literals
`rint`	✅	Native
`round`	✅	Hybrid	Float/double inputs route through the JVM codegen dispatcher; other types run natively
`sec`	✅	Native
`shiftleft`	✅	Native
`sign`	✅	Native
`signum`	✅	Native
`sin`	✅	Native
`sinh`	✅	Native
`sqrt`	✅	Native
`tan`	✅	Native
`tanh`	✅	Native
`try_add`	✅	—	Datetime/interval form falls back
`try_divide`	✅	—
`try_mod`	✅	—
`try_multiply`	✅	—
`try_subtract`	✅	—
`unhex`	✅	Native
`uniform`	✅	—	Constant-folded; literal arguments only (Spark 4.0+)
`width_bucket`	✅	Codegen dispatch

misc_funcs#

Function	Status	Implementation	Notes
`aes_decrypt`	✅	—	Routed through the JVM codegen dispatcher
`aes_encrypt`	✅	—	Routed through the JVM codegen dispatcher; nondeterministic IV by default
`assert_true`	🔜	—	Lowers to `RaiseError`, which falls back
`current_catalog`	✅	—	Resolved to a literal by the analyzer (`ReplaceCurrentLike`)
`current_database`	✅	—	Resolved to a literal by the analyzer (`ReplaceCurrentLike`)
`current_schema`	✅	—	Alias of `current_database`; resolved to a literal by the analyzer
`current_user`	✅	—	Resolved to a literal by the analyzer; same as `user`
`equal_null`	✅	—	Lowers to `<=>` (`EqualNullSafe`)
`hll_sketch_estimate`	✅	Native	Spark 4.0+ only; falls back by default, the native path is opt-in via allowIncompatible (details)
`hll_union`	✅	Native	Spark 4.0+ only; falls back by default, the native path is opt-in via allowIncompatible (details)
`is_variant_null`	🔜	—	Requires `VariantType` support
`monotonically_increasing_id`	✅	Native
`parse_json`	🔜	—	Requires `VariantType` support
`raise_error`	🔜	—	Raises a runtime error
`rand`	✅	Native	Seed must be a literal
`randn`	✅	Native	Seed must be a literal
`schema_of_variant`	🔜	—	Requires `VariantType` support
`schema_of_variant_agg`	🔜	—	Requires `VariantType` support
`session_user`	✅	—	Alias of `current_user`; resolved to a literal by the analyzer
`spark_partition_id`	✅	Native
`to_variant_object`	🔜	—	Requires `VariantType` support
`try_aes_decrypt`	✅	—	Routed through the JVM codegen dispatcher
`try_parse_json`	🔜	—	Requires `VariantType` support
`try_variant_get`	🔜	—	Requires `VariantType` support
`typeof`	✅	—	Foldable; resolved to a literal before Comet sees the plan
`user`	✅	—	Resolved to a literal by the Spark analyzer before reaching Comet
`uuid`	✅	Native
`variant_get`	🔜	—	Requires `VariantType` support

predicate_funcs#

Function	Status	Implementation	Notes
`!`	✅	Native
`<`	✅	Hybrid
`<=`	✅	Hybrid
`<=>`	✅	Hybrid
`=`	✅	Hybrid
`==`	✅	Hybrid
`>`	✅	Hybrid
`>=`	✅	Hybrid
`and`	✅	Native
`between`	✅	—
`ilike`	✅	—
`in`	✅	Hybrid
`isnan`	✅	Native
`isnotnull`	✅	Native
`isnull`	✅	Native
`like`	✅	Hybrid
`not`	✅	Native
`or`	✅	Native
`regexp`	✅	Hybrid	In-subset literal patterns run natively; other patterns route through the JVM codegen dispatcher (details)
`regexp_like`	✅	Hybrid	In-subset literal patterns run natively; other patterns route through the JVM codegen dispatcher (details)
`rlike`	✅	Hybrid	In-subset literal patterns run natively; other patterns route through the JVM codegen dispatcher (details)

string_funcs#

Function	Status	Implementation	Notes
`ascii`	✅	Native
`base64`	✅	Native
`bit_length`	✅	Native
`btrim`	✅	—
`char`	✅	Native
`char_length`	✅	Native
`character_length`	✅	Native
`chr`	✅	Native
`collate`	🔜	—	Spark collation (umbrella #2190)
`collation`	✅	—	Constant-folded to a literal (Spark 4.0+)
`concat_ws`	✅	Hybrid	Mixed string and array arguments run natively; all-foldable arguments use codegen dispatch
`contains`	✅	—
`decode`	✅	—
`elt`	✅	Codegen dispatch
`encode`	✅	—
`endswith`	✅	—
`find_in_set`	✅	Codegen dispatch
`format_number`	✅	Codegen dispatch
`format_string`	✅	Codegen dispatch
`initcap`	✅	Hybrid
`instr`	✅	Hybrid	Non-UTF8_BINARY collated input routes through the JVM codegen dispatcher; other input runs natively
`lcase`	✅	Hybrid
`left`	✅	Native
`len`	✅	Native
`length`	✅	Native
`levenshtein`	✅	Hybrid	Non-UTF8_BINARY collated input routes through the JVM codegen dispatcher; other input runs natively
`locate`	✅	Codegen dispatch
`lower`	✅	Hybrid
`lpad`	✅	Hybrid	String inputs use the native kernel with a column string and literal padding; literal strings and column padding use codegen dispatch. Binary inputs use codegen dispatch.
`ltrim`	✅	Hybrid	Non-UTF8_BINARY collated input with a trim string routes through the JVM codegen dispatcher; other input runs natively
`luhn_check`	✅	—	Native via `StaticInvoke` (tests: luhn_check.sql)
`mask`	✅	—	Routed through the JVM codegen dispatcher
`octet_length`	✅	Native
`overlay`	✅	Codegen dispatch
`position`	✅	Codegen dispatch
`printf`	✅	Codegen dispatch
`regexp_count`	✅	—	Rewrites to `size(regexp_extract_all(...))`; `regexp_extract_all` routes through the JVM codegen dispatcher by default
`regexp_extract`	✅	Native
`regexp_extract_all`	✅	Native
`regexp_instr`	✅	Codegen dispatch	Routed through the JVM codegen dispatcher
`regexp_replace`	✅	Hybrid
`regexp_substr`	✅	—	Rewrites to `nullif(regexp_extract(...), '')`; `regexp_extract` routes through the JVM codegen dispatcher by default
`repeat`	✅	Native
`replace`	✅	Hybrid	Native by default for a direct string column with a non-empty, valid UTF-8 search literal and either a replacement literal (both at most 64 KiB) or a replacement column (search at most 256 bytes and at most 8 MiB when broadcast per batch), under UTF8_BINARY collation; other cases use the JVM codegen dispatcher by default
`right`	✅	Native
`rpad`	✅	Hybrid	String inputs use the native kernel with a column string and literal padding; literal strings and column padding use codegen dispatch. Binary inputs use codegen dispatch.
`rtrim`	✅	Hybrid	Non-UTF8_BINARY collated input with a trim string routes through the JVM codegen dispatcher; other input runs natively
`soundex`	✅	Native
`space`	✅	Native
`split`	✅	Hybrid
`split_part`	✅	—	Spark 4.0+
`startswith`	✅	—
`substr`	✅	Native
`substring`	✅	Native
`substring_index`	✅	Hybrid	Non-UTF8_BINARY collated input routes through the JVM codegen dispatcher; other input runs natively
`to_binary`	✅	—	The hex form runs natively; the base64 and `utf-8` forms route through the JVM codegen dispatcher
`to_char`	✅	—
`to_number`	✅	Codegen dispatch
`to_varchar`	✅	—
`translate`	✅	Hybrid	Codegen dispatch by default: DataFusion’s `translate` iterates over Unicode graphemes (Spark uses code points) and substitutes U+0000 instead of treating it as a deletion sentinel, so the native path is opt-in via allowIncompatible
`trim`	✅	Hybrid	Non-UTF8_BINARY collated input with a trim string routes through the JVM codegen dispatcher; other input runs natively
`try_to_binary`	✅	—	Rewrites to `try_eval(to_binary(...))`, which routes through the JVM codegen dispatcher
`try_to_number`	✅	Codegen dispatch	Routed through the JVM codegen dispatcher
`ucase`	✅	Hybrid
`unbase64`	✅	Hybrid	Column or literal input runs natively; other child expressions, and the strict form used by `to_binary(str, 'base64')`, route through the JVM codegen dispatcher
`upper`	✅	Hybrid

struct_funcs#

Function	Status	Implementation	Notes
`named_struct`	✅	Native	Duplicate field names fall back
`struct`	✅	Native

url_funcs#

Function	Status	Implementation	Notes
`parse_url`	✅	Native
`try_parse_url`	✅	—	Spark 4.0+; runs natively and returns NULL for an invalid URL
`try_url_decode`	✅	—
`url_decode`	✅	—
`url_encode`	✅	—

window_funcs#

Window functions run via CometWindowExec, which is enabled by default. Aggregate window functions (count, min, max, sum, avg, first_value, last_value), ranking functions (row_number, rank, dense_rank, percent_rank, cume_dist, ntile), and value-shift functions (lag, lead, nth_value) are all wired in the window serde and execute natively. Statistical aggregates such as stddev, var_pop, corr, and covar_pop run natively as plain aggregations but fall back to Spark when used as window functions. A handful of frame shapes also fall back. See window function compatibility for the full list of supported functions, frames, and fallback cases.

Function	Status	Implementation	Notes
`cume_dist`	✅	—	via `CometWindowExec`
`dense_rank`	✅	—	via `CometWindowExec`
`lag`	✅	—	via `CometWindowExec`; non-literal default falls back (#4268)
`lead`	✅	—	via `CometWindowExec`; non-literal default falls back (#4268)
`nth_value`	✅	—	via `CometWindowExec`
`ntile`	✅	—	via `CometWindowExec`
`percent_rank`	✅	—	via `CometWindowExec`
`rank`	✅	—	via `CometWindowExec`
`row_number`	✅	—	via `CometWindowExec`

xml_funcs#

Function	Status	Implementation	Notes
`from_xml`	✅	—	Spark 4.0+
`schema_of_xml`	✅	—	Spark 4.0+
`to_xml`	✅	—	Spark 4.0+
`xpath`	✅	Codegen dispatch
`xpath_boolean`	✅	Codegen dispatch
`xpath_double`	✅	Codegen dispatch
`xpath_float`	✅	Codegen dispatch
`xpath_int`	✅	Codegen dispatch
`xpath_long`	✅	Codegen dispatch
`xpath_number`	✅	Codegen dispatch	Alias of `xpath_double`
`xpath_short`	✅	Codegen dispatch
`xpath_string`	✅	Codegen dispatch

Beyond SQL functions#

Comet also accelerates a number of Catalyst expressions that have no Spark SQL function name and therefore do not appear in the tables above. These arise from the DataFrame API, from SQL syntax other than function calls, or from the query optimizer. They include:

Operator and optimizer-injected expressions: runtime bloom-filter join probes (BloomFilterMightContain, BloomFilterAggregate), optimized IN sets (InSet), scalar subqueries (ScalarSubquery), and floating-point normalization (KnownFloatingPointNormalized).
Accessor expressions (subscript and field access, not functions): struct field access (col.field), array element access (arr[i]), and map value access (map[key]).
Internal decimal arithmetic: CheckOverflow, MakeDecimal, and UnscaledValue, which the analyzer inserts around decimal operations.
Missing-data predicates: AtLeastNNonNulls, used by DataFrame.na.drop, counts non-null and non-NaN values natively.
User-defined functions: Scala UDFs registered through the DataFrame or SQL API.
DataSource V2 catalog functions: Iceberg’s system functions bucket, truncate, years, months, days, and hours (for example system.bucket(16, id)) run natively; see Iceberg system functions. Other DataSource V2 catalog functions run in Spark.
Lowered built-ins: Spark lowers some built-in functions to StaticInvoke or Invoke calls. Those without a native mapping run through the JVM codegen dispatcher when their input and output types are supported.
Structural expressions: aliases, attribute references, literals, sort orders, and CASE WHEN.

This list is illustrative, not exhaustive: the per-function tables are not the complete set of expressions Comet can accelerate.

make_date evaluation#

With ANSI mode enabled, make_date(2024, 13, 1) raises an invalid-date error. If an input iterator yields a valid row followed by that invalid row, a LIMIT 1 can stop before evaluating the second date. A native projection can evaluate a whole batch before the consuming operator decides which rows it needs.

By default, Comet keeps affected operators in Spark to preserve skipped date evaluation. This applies to date projections below LIMIT, SAMPLE, FILTER, joins, mapPartitions, and Dataset.rdd consumers; date expressions inside joins, aggregates, and ordered limits; and projections below a window group limit. A parent projection can also defer a date output when an IF, COALESCE, or NULL-skipping operand does not need it. Comet preserves that evaluation across projections while allowing eager date consumers, such as YEAR(dt), to remain native when their operands are safe to evaluate. An enclosing conditional can still defer the extractor’s date input. These safeguards also preserve NULL short-circuiting when a date argument is computed in an earlier projection, including across the streamed side of a broadcast join. The materialized build side remains independent. These safeguards also cover required arguments, such as a computed year, on input rows that the join rejects before producing an output row. Selecting the date eagerly does not force conditionally used sibling expressions; those computations also retain Spark evaluation when needed.

With ANSI disabled, a direct date projection can remain native below these consumers when all its computations are safe to evaluate early. This includes make_date(2024, month, day) over input columns and years widened from TINYINT or SMALLINT. An unrestricted integer year still needs the safeguard: Spark’s conversion to Date32 can overflow independently of ANSI mode. Computed arguments, their deferred producers, and throwing sibling expressions retain their safeguards. Expression-level parent guards remain in effect. For example, if dt is a nullable date alias, unix_date(dt) + element_at(values, i) keeps its consuming operator in Spark when a NULL date can skip an invalid array index. The safe date-producing projection can remain native.

An unfiltered COUNT(make_date(...)) can remain native when its sole argument is nullable, because Spark checks that argument on every input row. Filtered counts and counts with multiple arguments retain the safeguard when they can skip date evaluation.

Only affected parts of the plan fall back. An ordinary native scan can remain native. An adjacent collect limit and date projection can also stay native: Comet caps the projection’s input before evaluating dates, so unread rows cannot raise errors. This requires a supported native local limit; otherwise the Spark safeguard remains. The cap does not bypass safeguards for a lower argument-producing projection or intervening filters and joins. Exchanges materialize their inputs and partition keys independently, so an outer limit does not force an eager date partition key to Spark. Local sorts and aggregates can also separate date evaluation from an outer limit.

Users who accept eagerly evaluating dates, projected arguments, and sibling expressions can set spark.comet.exec.makeDate.operatorFallback.enabled=false to allow native operators for these shapes. The default is true. Disabling it can expose invalid-date or argument errors that Spark would skip under ANSI mode. This setting does not change make_date’s invalid-date handling or the expression-level safeguards for skipped arguments and operands. For example, an empty IN predicate containing make_date still falls back when needed.