Expression Tuning#

Lambdas and Higher-Order Functions#

A higher-order function takes a lambda: filter(arr, x -> ...), transform, exists, forall, aggregate, zip_with and map_filter. How fast Comet runs one depends on whether it can run the lambda natively.

Lambda

How Comet runs it

filter(arr, x -> x IS NOT NULL) (what array_compact expands to), and filter(arr, x -> nvl(x, '') != '') on an array of numbers, dates or timestamps

Natively, as array_compact.

filter and exists with a one-parameter lambda whose body uses only natively supported expressions, over an array of primitive, decimal, string, binary, date or timestamp elements

Natively. The lambda runs once over all the elements of a batch, vectorized, instead of once per element.

Any other lambda, and every other higher-order function

Through the JVM codegen dispatcher. Spark’s interpreter evaluates the lambda for each element inside a Comet kernel, so expect Spark’s speed, not more.

exists runs natively under DataFusion’s array_any_match, which returns what Spark’s three-valued logic returns: NULL when no element matches but the lambda was NULL for some element. With spark.sql.legacy.followThreeValuedLogicInArrayExists=false, Spark returns false there, and exists stays on the dispatcher.

The dispatcher has no advantage over Spark for a lambda, because Spark also evaluates a lambda one element at a time. It adds the cost of handing each batch to the JVM and back. The rest of the query stays native either way.

To get the most out of Comet when a query uses lambdas:

  • Prefer a built-in array function to a lambda. Most of them run natively with no per-element evaluation, for example array_compact, array_remove, array_distinct and array_position. See the Supported Expressions tables for the caveats of each. array_compact(arr) returns what filter(arr, x -> x IS NOT NULL) does for any array, and array_remove(array_compact(arr), '') returns what filter(arr, x -> nvl(x, '') != '') does for an array of strings with the default collation.

  • Do not test a numeric array for nulls with a string comparison. With ANSI mode off, Spark casts every element of filter(int_arr, x -> nvl(x, '') != '') to a string to compare it with ''; with ANSI mode on the query does not analyze. Comet recognizes exactly this form, with either operand first, and runs it as array_compact, which never casts. A variation, such as another default value or an extra condition, runs as a general lambda and does cast every element. Write x IS NOT NULL, or call array_compact.

  • Keep the lambda body to natively supported expressions. A body that calls a regular expression function, a UDF or another lambda cannot be split from the dispatcher, so the whole lambda runs there. Use spark.comet.explain.codegen.enabled to see which expressions of a query went through the dispatcher.

  • Keep IN away from operations that can raise. Spark evaluates IN lazily: it stops at the first candidate that matches and evaluates none when the value is NULL. The native IN evaluates every candidate when one can raise, so a body that combines IN with an operation that can raise stays on the dispatcher. That covers a division, an addition or a cast under ANSI mode, which raise only in that mode, and element_at, which raises for the index 0 in any mode. AND, OR, CASE WHEN, IF and COALESCE are lazy natively too, so filter(arr, x -> x != 0 AND 100 / x > 1) runs natively.

  • Keep operations that can raise out of an exists lambda. Spark stops at the first element that matches, so under ANSI mode exists(array(1, 0), x -> 100 / x > 1) returns true without dividing by zero, but the native lambda evaluates every element. An exists lambda that contains any operation that can raise stays on the dispatcher, whether or not it uses AND, OR, IN or CASE WHEN.

  • Keep date and timestamp functions out of the lambda. Native date and timestamp functions support years up to about 262,143, while Spark supports them up to year 294,247. Beyond that, a cast between a date or timestamp and a string, date or timestamp fails natively (#7471), and year, month and the other date fields return NULL (#6843). So a lambda that applies a function to a date or timestamp stays on the dispatcher. Comparing one, testing it for NULL, choosing it with CASE WHEN, IF or COALESCE, and casting it to a number still run natively. A lambda with a try_cast that can make a NULL map key stays on the dispatcher too.

  • Capture only numeric, decimal, boolean, date or timestamp columns. DataFusion copies each column the lambda reads once per element of the array before it evaluates the lambda, so a captured string, binary, array, struct or map column is copied as many times as the array has elements. A lambda that reads one stays on the dispatcher.

  • Use a lambda of one parameter. filter(arr, (x, i) -> i > 0), which also receives the element’s index, runs through the dispatcher.

spark.comet.exec.nativeLambda.enabled=false makes Comet run every lambda through the dispatcher.

Example#

CometHigherOrderFunctionBenchmark runs filter over 1 million rows of ten-element arrays, in which every fifth element is NULL. These are the best times of the Spark baseline and of Comet on an Apple M-series laptop, with spark.comet.exec.nativeLambda.enabled off and at its default:

Query

Spark

Comet, nativeLambda off

Comet, default

filter(strs, x -> nvl(x, '') != '') (string array)

616 ms

547 ms

307 ms

array_remove(array_compact(strs), '') (same result)

587 ms

252 ms

252 ms

filter(ints, x -> x IS NOT NULL)

286 ms

124 ms

125 ms

filter(ints, x -> x > threshold), threshold is a column

346 ms

335 ms

110 ms

The same benchmark runs exists over 1 million rows of ten-element arrays, in which every fifth element is NULL (Spark 3.5, same laptop):

Query

Spark

Comet, nativeLambda off

Comet, default

exists(ints, x -> x > 90)

266 ms

234 ms

108 ms

exists(dbls, x -> x IS NULL OR x < 0 OR x > 90 OR x = 0)

266 ms

225 ms

94 ms

exists(ints, x -> x > threshold), threshold is a column

251 ms

196 ms

105 ms

size(dbls) < 1 OR exists(dbls, x -> x > 90)

355 ms

403 ms

110 ms

exists(strs, x -> x IS NULL OR x = '')

357 ms

183 ms

168 ms

Rewriting a lambda to built-in functions is still the fastest form, but a native lambda gets most of the way there.