Optimizing Scalar Expressions#
This guide describes how to optimize the native scalar expression implementations in the
datafusion-comet-spark-expr crate (native/spark-expr/). These are the per-row and per-batch
kernels that run inside DataFusion for Spark-compatible functions such as casts, string
functions, JSON encoders, and array functions.
The workflow is deliberate: measure first, keep the output bit-identical, and prove the win with a benchmark that covers the shapes where an optimization is most likely to backfire. Every step below exists because skipping it has produced either a no-op PR or a regression.
When to optimize#
Good candidates share these traits:
The function is called once per row or once per element, so allocation and branching costs multiply by batch size.
The hot path allocates (a
String, aVec, a slicedArrayRef, a compiledRegex) on every row, or dispatches an Arrow compute kernel per element.The function already has correctness tests, so behavior is pinned.
Do not optimize speculatively. If you cannot write a benchmark that shows a meaningful improvement, there is nothing to submit.
The workflow#
Read the current implementation in
native/spark-expr/src/. Identify the per-row cost: allocation, kernel dispatch, UTF-8 decoding, regex compilation, bitmap reads.Write or extend a criterion benchmark in
native/spark-expr/benches/(see below). Run it onmainto capture a baseline before changing any code.Apply the optimization, preserving exact semantics (see Correctness).
Run the existing unit tests for the function. They must pass unchanged. Output must be bit-identical to
main, including null placement and error behavior.Re-run the benchmark against the baseline. Confirm a meaningful speedup on the common shapes, and account for every shape that got slower (see Slower shapes need a justification).
Submit with the criterion output pasted into the PR description.
Writing a benchmark#
Each optimized expression should have a criterion benchmark under native/spark-expr/benches/
registered in native/spark-expr/Cargo.toml:
[[bench]]
name = "unhex"
harness = false
Model new benchmarks on an existing one such as benches/unhex.rs or benches/array_size.rs.
A benchmark builds representative Arrow arrays (typically 8192 rows, the default batch size),
wraps the call in black_box, and benches several input shapes:
fn criterion_benchmark(c: &mut Criterion) {
let size = 8192;
// ... build arrays for each shape ...
let mut bench = |name: &str, arr: &ArrayRef| {
let args = vec![ColumnarValue::Array(Arc::clone(arr))];
c.bench_function(name, |b| {
b.iter(|| black_box(spark_unhex(black_box(&args)).unwrap()))
});
};
bench("spark_unhex: all valid", &all_valid);
bench("spark_unhex: with nulls", &with_nulls);
bench("spark_unhex: long strings", &long_valid);
bench("spark_unhex: invalid inputs", &invalid);
bench("spark_unhex: mixed hex column", &mixed);
}
Cover the shapes that break optimizations#
The most common way a “faster” change is actually a regression is that it wins on the shape you looked at and loses on one you did not. Always include:
No nulls vs. sparse nulls vs. dense nulls. Fast paths that assume no nulls, or that batch contiguous non-null runs, can be slower than the base loop when nulls are dense.
Short vs. long values. Allocation savings shrink and copy costs grow as values get longer.
Valid vs. invalid inputs. Error and fallback paths have different costs.
ASCII vs. non-ASCII for string functions with an ASCII fast path.
Well-predicted vs. poorly predicted branches. Constant lengths or all-valid input against random lengths or values. Branchless code wins on the second and can lose on the first, and branchy code does the reverse.
Running benchmarks#
cd native
# capture the baseline on main
git checkout main
cargo bench --bench unhex -- --save-baseline main
# switch to your branch and compare
git checkout my-branch
cargo bench --bench unhex -- --baseline main
Criterion reports the percentage change and confidence interval per shape. A change inside the noise threshold is not an improvement.
Correctness is non-negotiable#
The output of the optimized function must be bit-identical to main for every input:
same values, same null buffer, same errors. Comet does not ship a differential fuzz harness in
this repo, so the existing unit tests are your correctness gate. If coverage is thin, add tests
(or run audit-comet-expression) before optimizing.
Recurring correctness traps:
Null slots must not raise errors. In ANSI/overflow paths, only apply the fallible conversion to non-null slots. Arrow’s
try_unarydoes this for you: a garbage value sitting under a null cannot raise a spurious overflow. Hand-rolled loops that read the value before checking the null bit will report errors Spark does not.Preserve the exact eval-mode semantics. For example, Legacy narrowing int casts keep the low-order bits (a wrapping
asconversion), while ANSI raises on overflow. These are different kernels, not a single path with a flag.Carry the null buffer through untouched when the values transform is infallible and one-to-one.
unary/try_unarydo this; a rebuild-from-iterator does not, and can drop or shift nulls.Zero-copy buffer reuse must respect offsets and slicing. Reusing an input array’s buffers is only valid when the layout genuinely matches the output.
Proven techniques#
These are the patterns that have produced real speedups in merged and in-flight Comet PRs. Reach for the lightest one that fits.
Technique |
What it replaces |
Example |
|---|---|---|
Vectorized Arrow kernels ( |
|
|
|
A builder loop that appends null on overflow, or |
int/float-to-decimal casts (28-50%): map overflow to null in one vectorized pass; for the ANSI throw-on-overflow variant, gate a rare element-wise rescan on an O(1) null-count check |
Zero-copy borrow with |
Allocating a new |
|
Zero-copy buffer reuse |
Copying every value into a fresh builder |
|
Preallocate builders to known size |
Repeated buffer growth/reallocation |
|
Compile-time lookup tables |
Per-element range matches / branching |
|
Cache compiled regex (thread-local, keyed by the constant arg) |
|
|
Read from the offset buffer directly |
|
|
Reuse Arrow |
Per-row |
|
Typed scans over flat values buffers + hash probe |
A per-element Arrow |
|
ASCII / byte-offset fast path |
|
|
|
Char-by-char |
|
Cross-cutting principles behind the table:
Hoist work that is constant across the batch out of the per-row loop (regex compilation, format setup, capacity that depends only on the whole column).
Avoid per-row heap allocation. Reuse a scratch buffer, or write directly into the output builder.
Prefer bulk operations (bulk-copy a run, one
extendper contiguous run) over per-element operations, but verify the dense-null case does not regress.Watch capacity hints. A
rows * rowscapacity hint (instead ofrows) silently over-allocates; fix these when you find them.
Slower shapes need a justification#
Optimize for the common case. A shape may get slower when the common case gets faster and the PR justifies it. For each shape that is meaningfully and reproducibly slower, the PR description gives the measured cost, how often real workloads reach that shape and why, and the gain on the common case. For example, an error path that fails the query runs at most once per partition, so a slower error path can be acceptable.
A shape that real workloads reach often is not an exception. A change that is 90% faster with no nulls but 30% slower with dense nulls costs the many real columns that have dense nulls, so either gate the fast path behind a runtime check that picks the right path per batch, or do not submit it. A Comet optimization PR was closed for exactly this reason. A claim that a shape is rare needs evidence about real workloads, not an assertion.
Record the performance audit#
Once a tuning change is merged, record it in the per-expression
Expression Audits so contributors can see which expressions have
already been optimized and avoid re-treading the same ground. Add a dated Performance (tuned ...)
line under the expression’s heading on the relevant category page (create the heading if the
expression has no audit entry yet):
## unhex
- Performance (tuned 2026-07-11, PR #4876): compile-time 256-entry hex lookup table plus
preallocated `BinaryBuilder`; up to 31% faster on long strings. Benchmark: `benches/unhex.rs`.
Name the technique, the measured speedup, the PR, the benchmark file, and any shape that got slower with its cost. Keep it to one line per tuning pass so the history stays scannable.
PR conventions#
Optimization PRs are small and follow a consistent shape:
Title:
perf: optimize `function_name` (Nx faster)with the headline speedup.Rationale: “Optimize existing expression.”
Changes: one or two sentences naming the specific technique and what per-row cost it removes.
Testing: “Existing tests.” plus the pasted criterion output for every shape, and any shape that got slower with its cost and justification.
Keep the diff focused on one expression. Do not bundle unrelated changes.
Follow the standard Pre-PR checklist: make format, build, and
cargo clippy --all-targets --workspace -- -D warnings.