Expression Tuning#
Lambdas and Higher-Order Functions#
A higher-order function takes a lambda: filter(arr, x -> ...), transform, exists, forall, aggregate,
zip_with and map_filter. How fast Comet runs one depends on whether it can run the lambda natively.
Lambda |
How Comet runs it |
|---|---|
|
Natively, as |
|
Natively. The lambda runs once over all the elements of a batch, vectorized, instead of once per element. |
Any other lambda, and every other higher-order function |
Through the JVM codegen dispatcher. Spark’s interpreter evaluates the lambda for each element inside a Comet kernel, so expect Spark’s speed, not more. |
exists runs natively under DataFusion’s array_any_match, which returns what Spark’s three-valued logic returns: NULL
when no element matches but the lambda was NULL for some element. With
spark.sql.legacy.followThreeValuedLogicInArrayExists=false, Spark returns false there, and exists stays on the
dispatcher.
The dispatcher has no advantage over Spark for a lambda, because Spark also evaluates a lambda one element at a time. It adds the cost of handing each batch to the JVM and back. The rest of the query stays native either way.
To get the most out of Comet when a query uses lambdas:
Prefer a built-in array function to a lambda. Most of them run natively with no per-element evaluation, for example
array_compact,array_remove,array_distinctandarray_position. See the Supported Expressions tables for the caveats of each.array_compact(arr)returns whatfilter(arr, x -> x IS NOT NULL)does for any array, andarray_remove(array_compact(arr), '')returns whatfilter(arr, x -> nvl(x, '') != '')does for an array of strings with the default collation.Do not test a numeric array for nulls with a string comparison. With ANSI mode off, Spark casts every element of
filter(int_arr, x -> nvl(x, '') != '')to a string to compare it with''; with ANSI mode on the query does not analyze. Comet recognizes exactly this form, with either operand first, and runs it asarray_compact, which never casts. A variation, such as another default value or an extra condition, runs as a general lambda and does cast every element. Writex IS NOT NULL, or callarray_compact.Keep the lambda body to natively supported expressions. A body that calls a regular expression function, a UDF or another lambda cannot be split from the dispatcher, so the whole lambda runs there. Use
spark.comet.explain.codegen.enabledto see which expressions of a query went through the dispatcher.Keep
INaway from operations that can raise. Spark evaluatesINlazily: it stops at the first candidate that matches and evaluates none when the value isNULL. The nativeINevaluates every candidate when one can raise, so a body that combinesINwith an operation that can raise stays on the dispatcher. That covers a division, an addition or a cast under ANSI mode, which raise only in that mode, andelement_at, which raises for the index 0 in any mode.AND,OR,CASE WHEN,IFandCOALESCEare lazy natively too, sofilter(arr, x -> x != 0 AND 100 / x > 1)runs natively.Keep operations that can raise out of an
existslambda. Spark stops at the first element that matches, so under ANSI modeexists(array(1, 0), x -> 100 / x > 1)returns true without dividing by zero, but the native lambda evaluates every element. Anexistslambda that contains any operation that can raise stays on the dispatcher, whether or not it usesAND,OR,INorCASE WHEN.Keep date and timestamp functions out of the lambda. Native date and timestamp functions support years up to about 262,143, while Spark supports them up to year 294,247. Beyond that, a cast between a date or timestamp and a string, date or timestamp fails natively (#7471), and
year,monthand the other date fields returnNULL(#6843). So a lambda that applies a function to a date or timestamp stays on the dispatcher. Comparing one, testing it forNULL, choosing it withCASE WHEN,IForCOALESCE, and casting it to a number still run natively. A lambda with atry_castthat can make aNULLmap key stays on the dispatcher too.Capture only numeric, decimal, boolean, date or timestamp columns. DataFusion copies each column the lambda reads once per element of the array before it evaluates the lambda, so a captured string, binary, array, struct or map column is copied as many times as the array has elements. A lambda that reads one stays on the dispatcher.
Use a lambda of one parameter.
filter(arr, (x, i) -> i > 0), which also receives the element’s index, runs through the dispatcher.
spark.comet.exec.nativeLambda.enabled=false makes Comet run every lambda through the dispatcher.
Example#
CometHigherOrderFunctionBenchmark runs filter over 1 million rows of ten-element arrays, in which every fifth element is
NULL. These are the best times of the Spark baseline and of Comet on an Apple M-series laptop, with
spark.comet.exec.nativeLambda.enabled off and at its default:
Query |
Spark |
Comet, nativeLambda off |
Comet, default |
|---|---|---|---|
|
616 ms |
547 ms |
307 ms |
|
587 ms |
252 ms |
252 ms |
|
286 ms |
124 ms |
125 ms |
|
346 ms |
335 ms |
110 ms |
The same benchmark runs exists over 1 million rows of ten-element arrays, in which every fifth element is NULL
(Spark 3.5, same laptop):
Query |
Spark |
Comet, nativeLambda off |
Comet, default |
|---|---|---|---|
|
266 ms |
234 ms |
108 ms |
|
266 ms |
225 ms |
94 ms |
|
251 ms |
196 ms |
105 ms |
|
355 ms |
403 ms |
110 ms |
|
357 ms |
183 ms |
168 ms |
Rewriting a lambda to built-in functions is still the fastest form, but a native lambda gets most of the way there.