agg_funcs Expression Audits#

Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide Spark Expression Support for current support status.

any#

  • Spark 3.4.3 (audited 2026-05-26): registered as a SQL alias of BoolOr, which extends RuntimeReplaceableAggregate with replacement = Max(child). Catalyst rewrites any(x) to max(x) before Comet sees the plan, so any is served by CometMax on a BooleanType column.

  • Spark 3.5.8 (audited 2026-05-26): identical to 3.4.3.

  • Spark 4.0.1 (audited 2026-05-26): identical to 3.4.3.

approx_percentile#

  • Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-07-02): ApproximatePercentile(child, percentageExpression, accuracyExpression) is a TypedImperativeAggregate backed by a Greenwald-Khanna PercentileDigest quantile summary with relative error 1.0 / accuracy. child accepts NumericType, DateType, TimestampType, TimestampNTZType, and interval types (all cast to double internally); percentage is a single literal or literal array in [0.0, 1.0]; accuracy is a positive literal (default 10000). NULL inputs are skipped; an empty or all-null group returns NULL. approx_percentile is a SQL alias for the primary function name percentile_approx.

  • CometApproxPercentile maps the byte, short, int, long, float, and double input forms to a native Greenwald-Khanna quantile summary port with the same insert/compress/merge/query algorithm and relative error, casting the result back to the input type. percentage and accuracy must be foldable literals, matching Spark. Date, timestamp, interval, and decimal inputs fall back to Spark.

avg#

  • Spark 3.4.3 (2026-05-26)

  • Spark 3.5.8 (2026-05-26): aggregate logic identical to 3.4.3

  • Spark 4.0.1 (2026-05-26): aggregate logic identical to 3.5.8; only QueryContext import path differs. YearMonthIntervalType and DayTimeIntervalType inputs (supported by Spark) fall back to Spark in Comet.

bit_and#

  • Spark 3.4.3 (2026-05-26)

  • Spark 3.5.8 (2026-05-26)

  • Spark 4.0.1 (2026-05-26)

median#

  • Spark 3.4.3 (audited 2026-06-24): Median(child) is a RuntimeReplaceableAggregate with replacement = Percentile(child, Literal(0.5)). Catalyst rewrites median(x) to percentile(x, 0.5) before Comet sees the plan, so it is served by CometPercentile.

  • Spark 3.5.8 (audited 2026-06-24): identical to 3.4.3.

  • Spark 4.0.1 (audited 2026-06-24): replacement becomes lazy val; semantics unchanged.

  • Spark 4.1.1 (audited 2026-06-24): identical to 4.0.1.

percentile#

  • Spark 3.4.3 (audited 2026-06-24): Percentile(child, percentageExpression, frequencyExpression, ..., reverse) over PercentileBase. Exact percentile using index = p * (n - 1) linear interpolation, NULL inputs skipped, empty/all-null group returns NULL. CometPercentile maps the single-literal-percentage, default-frequency, numeric-input, ascending form to DataFusion’s percentile_cont (same interpolation). Array-of-percentages, a non-default frequency argument, descending order, and interval inputs fall back to Spark.

  • Spark 3.5.8 (audited 2026-06-24): ordering centralized via PhysicalDataType.ordering; behavior identical to 3.4.3.

  • Spark 4.0.1 (audited 2026-06-24): adds PercentileCont/PercentileDisc builders and SupportsOrderingWithinGroup, enabling percentile_cont(p) WITHIN GROUP (ORDER BY col), which rewrites to Percentile(col, p, reverse). The ascending form runs natively; the DESC form sets reverse = true and falls back to Spark because the native percentile_cont always interpolates in ascending order.

  • Spark 4.1.1 (audited 2026-06-24): identical to 4.0.1.

  • CometPercentile reports Incompatible for the otherwise-supported form because DataFusion’s percentile_cont quantizes the interpolation weight to 6 decimal places (INTERPOLATION_PRECISION = 1e6), so a deeply-interpolated value can differ from Spark by up to roughly (upper - lower) * 1e-6. The native path is opt-in via spark.comet.expression.Percentile.allowIncompatible=true (#4719).