DataFusion Comet 1.2.0 Changelog

This release consists of 396 commits from 40 contributors. See credits at the end of this changelog for more information.

Fixed bugs:

• fix: reject a file without field ids at any depth whether or not id matching is on #6116 (dwsmith1983)

• fix: decline native Iceberg writes with a custom location provider #6216 (liupoyi-1031)

• fix: report task input metrics after the native iterator closes and add to Spark’s counters #5880 (dwsmith1983)

• fix: Native S3 scan on EKS/IRSA turns a transient STS throttle into a hard 403 storm #6025 (parthchandra)

• fix: log partial memory grants at DEBUG and drop the memory usage dump #6269 (andygrove)

• fix: prevent libhdfs thread destructor use-after-free #5036 (peterxcli)

• fix: return a native plan’s memory before its Spark task ends #6261 (andygrove)

• fix: native ansi-errors raised as spark error #5169 (peterxcli)

• fix: credit co-authors of a PR in the generated changelog #6358 (andygrove)

• fix: validate map constructor row lengths and null short-circuiting #5846 (sunchao)

• fix: match Spark ordering in native array extrema #5403 (sunchao)

• fix: label native timestamp_seconds results as UTC timestamps #6341 (andygrove)

• fix: keep the input’s timezone label on native date_trunc results #6345 (andygrove)

• fix: validate spark.comet.shuffle.jvm.batchSize and spark.comet.exec.memoryPool #6287 (andygrove)

• fix: reject non-default collations in all sort keys #6206 (LinSimon-901101)

• fix: zero sliced boolean offsets at every level before exporting to the JVM #6339 (andygrove)

• fix: count pool overcommit as untracked memory in the native memory usage log #6271 (andygrove)

• fix: install Comet’s cache serializer only when Comet and native execution are enabled #6360 (andygrove)

• fix: fall back from the native CSV scan for timestamps outside UTC #6349 (andygrove)

• fix: key each scan’s planning data by its plan node in a native block #6297 (dwsmith1983)

• fix: derive the native memory limit the way YARN and Kubernetes size executor containers #6375 (andygrove)

• fix: route nested DATE to numeric casts in structs and maps through the codegen dispatcher #6319 (andygrove)

• fix: match iceberg-java for Iceberg partitions past year 262142 in the native writer #6239 (andygrove)

• fix: count each reused exchange once in extended explain #6379 (andygrove)

• fix: normalize session timezone IDs before passing them to native code #6351 (andygrove)

• fix: count every hash-based JVM shuffle batch as shuffle bytes written #6369 (andygrove)

• fix: release input Arrow streams that native never takes #6372 (andygrove)

• fix: make benchmark cases over ShortType columns exercise Comet #6376 (andygrove)

• fix: fail a native plan whose producer task is cancelled before its stream ends #6362 (andygrove)

• fix: relabel CASE and COALESCE timestamp branches instead of panicking #6347 (andygrove)

• fix: warn about unreleased native memory only after a task’s last native plan closes #6374 (andygrove)

• fix: cap spark.comet.shuffle.jvm.batchSize at spark.comet.batchSize where it is read #6366 (andygrove)

• fix: warn when native and JVM timezone databases differ #6353 (andygrove)

• fix: describe CometInMemoryTableScan by the scan it replaces in EXPLAIN #6453 (andygrove)

• fix: remove nested planner placeholders after recursive conversion #6439 (sunchao)

• fix: fall back to Spark for RANK and DENSE_RANK limits over nested floating-point keys #6468 (andygrove)

• fix: make adaptive partial aggregation opt-in with spark.comet.exec.aggregate.skipPartial.enabled #6474 (andygrove)

• fix: fall back when struct field names collide case-insensitively #6326 (comphead)

• fix: reject casts involving non-default collated strings #5302 (stantheman0128)

• fix: get_json_object returns first value for duplicate keys to match Spark #4971 (u70b3)

• fix: build with Rust 1.99, which deprecates the legacy f64 constants and fetch_update #6507 (andygrove)

• fix: fall back to Spark for _metadata.file_block_start and file_block_length #6510 (andygrove)

• fix: report a cached relation’s decoded size to the planner, not its compressed size #6412 (andygrove)

• fix: Route collated instr, substring_index, trim, greatest and least through the codegen dispatcher #6480 (viirya)

• fix: keep Spark’s cache scan for a relation whose cached plan records observed metrics #6421 (andygrove)

• fix: load the bundled native library only once per class loader #6493 (zhangfengcdt)

• fix: match Iceberg’s rounding for pre-1970 timestamps in native years/months/days/hours #6456 (andygrove)

• fix: cast native IF branches to a common type like CASE WHEN #6458 (andygrove)

• fix: fall back to Spark for regr_* aggregates until their merge matches Spark #6451 (andygrove)

• fix: defer Parquet conversion errors until a row group is decoded, as Spark does #6515 (andygrove)

• fix: publish metrics on the update interval for native blocks without a JVM input #6314 (andygrove)

• fix: match Spark’s whitespace trimming in casts from string to timestamp and timestamp_ntz #6445 (andygrove)

• fix: run reverse of a binary value through the codegen dispatcher #6461 (andygrove)

• fix: match Spark’s float semantics in array_remove and sort_array #6518 (andygrove)

• fix: fall back from the native Iceberg scan when a nested field was added or renamed #6543 (andygrove)

• fix: dispatch StaticInvoke and Invoke only into Spark’s own classes #6542 (andygrove)

• fix: fall back for Iceberg writes with float or double identity partitions #6531 (zhangfengcdt)

• fix: match Spark’s float ordering for sort and window keys nested in arrays and structs #6475 (andygrove)

• fix: check accelerated mapInArrow output against the declared schema #6365 (andygrove)

• fix: gate array distinct and union signed-zero semantics by Spark version #5750 (peterxcli)

• fix: hash NaN the way Spark does in hash, xxhash64 and native shuffle #6413 (andygrove)

• fix: return -0.0 for signum(-0.0), as Spark does #6556 (zhangfengcdt)

• fix: validate release build ref before building #6571 (andygrove)

• fix: draw the cached plan below CometInMemoryTableScan in EXPLAIN #6574 (comphead)

• fix: stop the leak of Arrow memory from struct outputs of the codegen dispatcher #6552 (peterxcli)

• fix: keep Spark’s cache format when Kryo would reject Comet’s or Comet disables itself #6537 (andygrove)

• fix: add native metrics from every plan a task runs instead of keeping the last #6416 (dwsmith1983)

• fix: keep a folded single-entry map literal with a null key on Spark #6585 (Smallfu666)

• fix: match Spark’s float ordering in min, max, greatest and least #6457 (andygrove)

• fix: convert native ReadAncientDatetime errors to SparkUpgradeException #6559 (dwsmith1983)

• fix: choose get_json_object number-limit behavior from the runtime Jackson version #6600 (andygrove)

• fix: align wide-decimal ANSI overflow value with Spark #5276 (sam-1112)

• fix: draw the cached plan below CometInMemoryTableScan in the SQL tab and event log #6577 (comphead)

• fix: preserve DPP filters during transition revert #6137 (sam-1112)

• fix: close Iceberg cleanup ownership gap during JVM handoff #6582 (unikdahal)

• fix(iceberg): preserve Spark 4.2 write transactions #6587 (unikdahal)

• fix: read the Iceberg write gate’s storage scheme the same way the native factory does #6502 (0lai0)

• fix: keep parquet view types out of the shuffle benchmark input #5804 (peterxcli)

• fix: make the native Iceberg per-I/O timeout configurable #6649 (comphead)

• fix: refresh stale shuffle scans when AQE prepares a stage, not while it re-plans #6646 (andygrove)

• fix: compact already-sliced batches before the native Iceberg writer counts NaNs #6238 (andygrove)

• fix: ship NOTICE.txt as META-INF/NOTICE in the comet-spark jar and sources jar #6657 (andygrove)

• fix: match Spark ANSI integral overflow errors (#6217) #6249 (LinSimon-901101)

• fix: fall back for RANGE frames over nested keys that can hold a null #6684 (andygrove)

• fix: decline native Iceberg writes through a void partition field whose source column was dropped #6677 (andygrove)

• fix: let a spilled final aggregate read its spill files back past its memory share #6544 (andygrove)

• fix: drop values under NULL parents before the native Iceberg writer counts NaNs #6658 (andygrove)

• fix: have native shuffle read a converted child’s Arrow stream #6689 (andygrove)

• fix: coalesce shuffle partitions under Comet unions the way Spark does #6459 (andygrove)

• fix: fall back for nested sort keys whose null elements Spark orders differently #6683 (andygrove)

• fix: gate unsupported Iceberg S3 write settings #6441 (sam-1112)

• fix: rethrow the exception a JVM input throws instead of a CometNativeException #6243 (andygrove)

• fix: initialize dispatched kernels with the partition the native plan computes #6700 (comphead)

• fix: restore Spark write execs when reverting transition-heavy stages #5957 (sam-1112)

• fix: count the days partition transform in UTC #6348 (andygrove)

• fix: normalize float operands in the native comparison builder #6447 (andygrove)

• fix: keep a Spark scan unconverted when the plan reads input_file_name #6703 (kazuyukitanimura)

• fix: load the Iceberg storage scheme lists from the native factory #6065 (dwsmith1983)

• fix: Route collated array element comparisons through the codegen dispatcher #6471 (viirya)

• fix: match Spark 4.2 date_trunc overflow for SECOND and MILLISECOND #6740 (andygrove)

• fix: make Iceberg delete-file reflection failures fatal #5515 (unikdahal)

• fix: honor Spark’s commit protocol in Spark 3.x native writes #4746 (peterxcli, andygrove)

• fix: retry a shuffle page allocation that Spark failed after dropping the task’s entry #6403 (dwsmith1983)

• fix: truncate per-row date_trunc formats with the literal-format kernel #6354 (andygrove)

• fix: don’t let S3A committer, delete and read defaults block native Iceberg writes #6808 (andygrove)

• fix: run InsertIntoHadoopFsRelationCommand’s post-write steps in the Spark 3.x native writer #6821 (viirya)

• fix: retry a native acquire that Spark failed after dropping the task’s entry #6310 (dwsmith1983)

• fix: check native UDF types at registration and explain planning-time evaluation #6777 (andygrove)

• fix: match Spark percentile ordering for negative NaNs #6829 (rich7420)

• fix: resolve Azure ABFS authentication through Hadoop and keep ambient credentials out #6059 (dwsmith1983)

• fix: fall back from native V2 scans when the plan reads input_file_name #6754 (zhangfengcdt)

• fix: accept all BMP Unicode decimal digits in cast(string as decimal) #6841 (viirya)

• fix: preserve CASE WHEN struct field names from the first THEN branch #6827 (pingzh)

• fix: keep one row per key in a native PartialMerge that runs short of memory #6867 (andygrove)

• fix: use Spark-compatible shuffle for wide decimal hash keys #6005 (sunchao, andygrove)

• fix: [spark34] preserve empty Iceberg partitions after dynamic pruning #6814 (LinSimon-901101)

• fix: restore localReads assertion in AdaptiveQueryExecSuite diffs #6382 (mizulun)

• fix: do not resolve DPP subqueries when AQE reads a native scan’s partitioning #6268 (dwsmith1983)

• fix: skip an array or map function’s later arguments where the array or map is NULL #6716 (andygrove)

• fix: share the missing task entry retry and stop retrying it after the task has ended #6813 (viirya)

• fix: return an empty array from CASE WHEN for an empty batch #6763 (viirya)

• fix: compare greatest and least struct arguments by position #6760 (viirya)

• fix: order nulls inside nested values first in least #6761 (viirya)

• fix: handle truncation operands in scalar evaluation and rewrites #6831 (nat-openai)

• fix: match Spark statistical aggregate updates and merges #6076 (rich7420)

• fix: make Comet’s in-memory cache opt-in again #6963 (andygrove)

• fix: keep the values of sliced batches when coalescing a broadcast #6900 (comphead)

• fix: compact oversized nested list_extract buffers #6498 (rich7420)

• fix: raise DIVIDE_BY_ZERO from native corr on a zero denominator in ANSI mode #6830 (kazuyukitanimura, rich7420)

• fix: report row contents to native write statistics trackers #6383 (LinSimon-901101)

• fix: fall back to Spark for decimal AVG cases that return wrong results #7017 (andygrove)

• fix: do not fail planning on a cast of a literal that fails to evaluate #6826 (viirya)

• fix: read each native scan partition through the object store its files live in #6753 (dwsmith1983)

• fix: compare a Remainder divisor with zero in Spark’s ordering without ANSI #6896 (andygrove)

• fix: include try_add suggestion in ANSI SUM overflow error #6732 (DeviousCardi)

• fix: give each nondeterministic dispatched expression its own kernel #6712 (dwsmith1983)

• fix: honor subexpression elimination off in the native null short-circuit #6986 (dwsmith1983)

• fix: read fs.comet.libhdfs.schemes for the scan rule from the Hadoop conf #6838 (dwsmith1983)

• fix: keep unconverted DPP filters in a scan’s canonical form #6270 (dwsmith1983)

• fix: fall back to Spark for NOT IN over an ordered streamed side #6810 (dwsmith1983)

• fix: stop building maps natively when the key type holds a FLOAT or DOUBLE #6992 (andygrove)

• fix: return errors instead of panicking in native planner #7087 (0lai0)

• fix: handle NullType struct fields in JVM columnar shuffle #7099 (jadenfix)

• fix: size the native runtime for standalone executors without spark.executor.cores #6368 (andygrove)

• fix: fall back to Spark when Iceberg reports an output ordering #7043 (parthchandra, andygrove)

• fix: build release jars from the same commit as the native libraries #6377 (andygrove)

• fix: count grouped integer SUM state in the aggregate’s memory reservation #6370 (andygrove)

• fix: preserve Parquet root field IDs after nested pruning #7081 (LinSimon-901101)

• fix: serialize large-offset string and binary vectors with 32-bit offsets #6371 (andygrove)

• fix: keep a streamed side sorted outside the native plan in order through outer hash joins #6835 (andygrove)

• fix: treat -0.0 and 0.0 as one value in arrays_overlap on Spark 4.0.5, 4.1.4 and 4.2.1 #6989 (andygrove)

• fix: preserve Spark nested null ordering in native sort #6968 (Weijun-H)

• fix: fall back when an Iceberg equality delete is keyed on a nested field #7122 (yunchipang)

• fix: fall back for duplicate from_json fields #6651 (Samearth17)

• fix: evaluate the array_append item on rows whose array is NULL #7025 (kazuyukitanimura)

• fix: fall back to Spark for negative-scale decimal casts and arithmetic #5050 (0lai0, andygrove)

• fix: extract bundled native libraries into private directories #6987 (manuzhang)

• fix: preserve Spark AVG buffer and overflow semantics #5420 (sunchao)

• fix: log native panic backtraces instead of attaching them to the exception #7042 (viirya)

• fix: report output rows and time of the native window operator #7140 (comphead)

• fix: [branch-1.2] keep Text build sides on Spark by default and add Text conversion configs (#7123) #7205 (andygrove)

• fix: [branch-1.2] don’t treat planner-inserted casts as Parquet schema conversions (#7159) #7262 (snmvaughan)

• fix: [branch-1.2] match SPARK-58821 and SPARK-60118 in sequence and string to decimal cast (#7171) #7263 (snmvaughan)

• fix: [branch-1.2] fall back to Spark when the driver cannot use the ABFS authentication (#7221) #7285 (andygrove)

• fix: [branch-1.2] spill a native PartialMerge whose input is sorted on some of its keys (#7222) #7286 (andygrove)

Performance related:

• perf: optimize string contains function #5322 (kazantsev-maksim)

• perf: cache Iceberg FileIO per executor instead of building one per task #6106 (mixermt)

• perf: copy off-heap strings directly into Arrow #6281 (pingzh)

• perf: reuse Variant reconstruction buffers #6342 (peterxcli)

• perf: memoize Comet-subtree check in EliminateRedundantTransitions #5213 (andygrove)

• perf: optimize native CASE WHEN and IF (up to 12x faster) #6350 (andygrove)

• perf: optimize levenshtein function #5042 (kazantsev-maksim)

• perf: read the shuffle natively in operators AQE reuses from the initial plan #6547 (dwsmith1983)

• perf: classify string-to-timestamp shapes in one byte scan (up to 5.6x faster) #5130 (andygrove)

• perf: write Spark columns to Arrow in bulk in CometSparkToColumnarExec #6566 (andygrove)

• perf: prune unused filter output columns #6396 (rich7420)

• perf: speed up string, map and array conversion in JVM columnar shuffle #6604 (andygrove)

• test: cover IF branch selection and short-circuiting #6397 (rich7420)

• perf: fuse Comet cache vector reads into Spark codegen #5859 (peterxcli)

• perf: choose dictionary encoding like iceberg-java in the native Iceberg writer #6241 (andygrove)

• perf: skip Spark’s encoders for boxed primitive Scala UDF values in the codegen dispatcher #6714 (andygrove)

• perf: key the codegen dispatcher’s kernel cache on a digest of the expression #6709 (andygrove)

• perf: convert Spark’s unsafe rows to Arrow straight from their memory #6730 (andygrove)

• perf: keep the sort-merge join when the forced hash join’s build side is too large #6384 (dwsmith1983)

• perf: compare floats in Spark’s order without normalizing copies of the operands #6691 (andygrove)

• perf: run Spark’s null guard for a Scala UDF inside the codegen kernel #6715 (andygrove)

• perf: bulk-append long nullable primitive arrays in SparkUnsafeArray conversion #6824 (andygrove, sandugood)

• perf: skip DecimalPrecision.promote’s rewrite when there is no decimal arithmetic #5216 (andygrove)

• perf: integral sequence optimisation #6833 (nat-openai)

• perf: generate the columnar-to-row projection once per executor instead of per partition #6858 (andygrove)

• perf: read shuffle blocks in 64 KiB pieces instead of through Channels.newChannel #6805 (comphead)

• perf: presize collections and reuse buffers in native loops #6861 (comphead)

• perf: read native-written Parquet footers without parsing Hadoop’s configuration #6923 (andygrove)

• perf: avoid duplicate Hadoop configs in Iceberg write detection #6925 (andygrove)

• perf: generate the in-memory cache reader’s class once per executor #6860 (andygrove)

• perf: store shuffle offsets as lengths with zstd #6943 (andygrove)

• perf: read only the nested fields Spark keeps in the native Iceberg scan #6725 (comphead)

• perf: make the native Iceberg writer’s dictionary pre-pass 1.8x to 3.9x faster #6937 (andygrove)

• perf: reuse validity buffers and use unary kernels in native expressions #6953 (comphead)

• perf: optimize JVM columnar-to-row conversion #5496 (peterxcli)

• perf: start bypass shuffle partition writers on small pages that grow #7059 (andygrove)

• perf: write every partition of a JVM shuffle spill in one native call #7065 (andygrove)

• perf: create a native plan’s spill directories on its first spill #7066 (andygrove)

• perf: vectorize int, decimal, float and boolean casts to timestamp #6969 (Ersa-tech)

• perf: sort JVM shuffle spill pointers in the sorter’s scratch space #7125 (viirya)

• perf: load the next Parquet file’s footer and page index while the current file is read #7094 (andygrove)

• perf: default the native Iceberg scan to two concurrent data files #7132 (andygrove)

• perf: share one decoded copy of a broadcast among the tasks of an executor #7016 (andygrove)

Implemented enhancements:

• test: report which writer ran each Iceberg write in the Iceberg CI jobs #6155 (andygrove)

• test: wait for Parquet write plan callbacks #6108 (rich7420)

• feat: add opt-in local TopK fusion for native Parquet scans #6181 (pingzh)

• feat: route timestamp_seconds decimal, byte and short input through codegen dispatch #6204 (0lai0)

• chore: add branch protection for branch-1.0 and branch-1.1 #6267 (andygrove)

• test: cover native Iceberg mid-write retry cleanup #6111 (sam-1112)

• feat: built-in S3 credential provider adapters for the native Parquet scan #6023 (parthchandra)

• feat: report JVM Arrow memory in the executor memory usage log #6250 (andygrove)

• test: cover matching nested nulls in collect aggregates #5970 (sunchao)

• test: fix flaky mixed field id directory test on Spark 3.4 and 3.5 #6312 (andygrove)

• feat: support Spark HyperLogLog sketch functions (hll_sketch_agg, hll_union_agg, hll_sketch_estimate, hll_union) #4802 (andygrove)

• build: enable Spark SQL tests for released Spark 4.2.0 #4950 (andygrove)

• feat: support divide_dt_interval with codegen dispatch #4901 (peterxcli)

• chore: start 1.2.0 development #6311 (andygrove)

• test: compare in-memory caches with independent answers #6262 (LinSimon-901101)

• test: cover native Iceberg write handoff cleanup #6117 (sam-1112)

• chore: Fix Scala code warnings #5876 (athlcode)

• test: backport the upstream fix for a flaky Iceberg 1.9.1 rewrite test #6242 (andygrove)

• ci: retry dependency downloads in the Spark SQL test jobs and the Iceberg Gradle wrapper #6394 (andygrove)

• ci: rebalance the Spark SQL test shards #6393 (andygrove)

• ci: raise the merge queue’s build concurrency from 2 to 4 #6391 (andygrove)

• ci: stop main’s cargo build cache from being evicted #6392 (andygrove)

• refactor: move native -0.0 and NaN handling into one float_semantics module #6400 (andygrove)

• build: bump the Spark 4.2 test diff to Comet 1.2.0-SNAPSHOT #6398 (andygrove)

• test: check Iceberg DPP pruning by planned file tasks #6280 (dwsmith1983)

• ci: keep a macOS native build cache on main #6395 (andygrove)

• test: cover integer bounds in cache pruning tests #6381 (LinSimon-901101)

• chore: remove AlignedArrowStreamReader and fix the Native to JVM FFI docs #6364 (andygrove)

• feat: push local TopK thresholds into Parquet readers #6263 (pingzh)

• feat: write the executor memory usage log to the Spark event log #6401 (comphead)

• refactor: separate JNI-free core logic and error classification at the native boundary #6435 (viirya)

• feat: admit string maps in Spark-to-Comet conversion #6036 (sunchao, jeffw13)

• test: cover Iceberg driver commit cleanup #6119 (sam-1112)

• refactor: move the remaining PhysicalExpr impls from core into spark-expr #6450 (andygrove)

• build: bump semanticdb to 4.13.10 so scalafix runs on Spark 4.1 and 4.2 #6448 (andygrove)

• test: cover booleans in native explode output past the first batch #6473 (andygrove)

• test: cover typed Dataset filters over sliced aggregate batches #6497 (rich7420)

• chore: move native write configs under spark.comet.write #6494 (zhangfengcdt)

• refactor: move the self-contained operators from core into a new operators crate #6503 (andygrove)

• refactor: move write planning into dedicated module #6529 (andygrove)

• feat: wire native existence join support #4587 (coderfender)

• test: sweep Spark’s -0.0 and NaN semantics across operators and float expressions #6521 (andygrove)

• feat: reuse S3 credentials until shortly before their reported expiry #6509 (snmvaughan)

• test: run Comet in Spark SQL and Iceberg suites that build their own sessions #6415 (andygrove)

• feat: record why Spark scans a relation cached in Comet’s format when Comet is off #6538 (andygrove)

• refactor: move shuffle planning into dedicated module #6535 (andygrove)

• ci: grant label runs the contents: write the docs job requests #6409 (andygrove)

• feat: per-location credentials for native Iceberg reads and writes #6478 (snmvaughan)

• feat: add a native Comet version of RangeExec #6553 (andygrove)

• feat: support multiply_ym_interval with YearMonth interval codegen dispatch #4715 (peterxcli)

• feat: add native distinct-combined collect_list aggregate support #4727 (peterxcli)

• test: add seeded fuzz coverage for PyArrow UDFs #6074 (LinSimon-901101)

• feat: add native support for MergeRowsExec (row-level MERGE INTO) #5318 (unikdahal)

• feat: give each default Spark-to-Arrow conversion its own spark.comet.convert config #6602 (andygrove)

• test: use the renamed native write flag in the Iceberg 1.11 WAP-branch suite #6644 (andygrove)

• feat: support Spark 4.1 TIME type and expressions via codegen dispatch #4821 (andygrove)

• refactor: move the remaining JNI-free entry point logic into core functions #6534 (viirya)

• feat: support native MergeSummary on Spark 4.1+ #6640 (unikdahal)

• ci: check licenses on release source tarball #6639 (andygrove)

• test: cover sequence’s per-batch element limit and the batch size remedy #6302 (dwsmith1983)

• ci: drop non-existent suites from the Linux and macOS test matrices #6099 (ErikBPF)

• feat: run the operators above a typed Dataset operation natively #6564 (andygrove)

• chore: update the NOTICE copyright year to 2024-2026 #6675 (manuzhang)

• chore: Add contributor sentiment to contribute #6688 (comphead)

• feat: support SortAggregateExec #4565 (andygrove)

• feat: custom Rust UDFs via arrow-ffi [experimental] #4459 (andygrove)

• test: keep the Iceberg job-abort test’s wait off Comet’s Tokio workers #6699 (andygrove)

• chore: raise the native rust-version to 1.95, which iceberg-rust requires #6690 (manuzhang)

• feat: add a spark.comet.convert config for RowDataSourceScanExec #6662 (andygrove)

• feat: support native Iceberg writes to format-version 3 tables #6672 (andygrove)

• feat: convert OneRowRelation to Arrow by default #6666 (andygrove)

• feat: charge native write buffers to the task’s memory pool #6247 (andygrove)

• test: move mixed-type shift coverage to SQL fixtures #6652 (rich7420)

• feat: Use DataFusion date_trunc for scalar-format timestamp truncation #5956 (sam-1112)

• chore: add benches for remaining comet native kernels #6648 (coderfender)

• chore: update documentation links for 1.1.0 release #6736 (andygrove)

• feat: add a spill compression codec config, defaulting to lz4 #6698 (comphead)

• feat(iceberg): plan WriteDelta in the split write path and keep it on the JVM DeltaWriter #6693 (unikdahal)

• feat: use native shuffle for row input by converting it to Arrow #6607 (andygrove)

• test: preserve regex escapes in SQL fixtures #6745 (rich7420)

• test: enable SPARK-57298 collect_set tests #5228 (peterxcli)

• test: add optimizer controls and config isolation to SQL fixtures #6731 (rich7420)

• chore: move CometCollationSuite to spark-4.1+ test shim #4864 (0lai0)

• test: drain the listener bus before registering Iceberg test listeners #6809 (viirya)

• feat: run nullable nested float sort keys natively in strict floating-point mode #6735 (andygrove)

• feat: enable Comet’s in-memory cache by default #5634 (andygrove)

• feat: run array_contains on float elements natively with Spark’s equa… #6599 (zhangfengcdt)

• refactor: move the plan entry point logic into core functions #6695 (viirya)

• refactor: remove unused cast compatibility flag #6135 (sam-1112, andygrove)

• test: move bit_count boundary coverage to SQL fixtures #6655 (rich7420)

• test: cover container field ids next to INT96 timestamps in the native Parquet scan #6862 (dwsmith1983)

• feat: filter decoded probe batches before intermediate joins #6433 (sunchao)

• test: count the partitions and batches the native cache scan reads #6836 (andygrove)

• feat: add lz4 as an opt-in codec for Comet’s in-memory cache #6859 (andygrove)

• chore: re-enable unused-parameter warnings in strict profile #6846 (rich7420)

• test: migrate required-child hash regressions to SQL fixtures #6864 (rich7420)

• test: fail Comet tests when a native plan mislabels a timestamp’s timezone #6868 (andygrove)

• test: migrate regex extraction and position coverage to SQL fixtures #6882 (rich7420)

• refactor: stop normalizing float divisors now that percentile_approx orders NaN as Spark does #6866 (andygrove)

• test: migrate scalar hash coverage to SQL fixtures #6881 (rich7420)

• test: migrate Java RLIKE semantics and batch coverage to SQL fixtures #6863 (rich7420)

• test: add SQL error-class parity checks #6650 (rich7420)

• test: run the S3 suites against an available MinIO image #6887 (dwsmith1983)

• build: stop publishing comet-common, the parent pom and comet-spark-integration #6888 (andygrove)

• feat: close fanout partitions early when the memory pool refuses the native Iceberg writer #6773 (andygrove)

• test: expect Spark 3.5’s empty LocalTableScan in the single-split DPP join test #6902 (dwsmith1983)

• feat: add native Delta Lake scan contrib module (page/row-group pruning) #5365 (dwsmith1983, sunchao, andygrove)

• test: Deflake SQLAppStatusListenerMemoryLeakSuite #6905 (LinSimon-901101)

• feat: enable native Iceberg writes by default #6664 (andygrove)

• test: add an end-to-end Iceberg read, repartition, and write benchmark #6910 (andygrove)

• ci: retry the scalafix lint when Maven cannot resolve the plugin #6061 (andygrove)

• feat: run min and max over string and binary columns natively #6927 (comphead)

• build: drop the compiled maven-wrapper.jar by switching to the script-only wrapper #6678 (manuzhang)

• feat: run global aggregates with no grouping keys or functions natively #6899 (andygrove)

• feat: normalize floats in native array_distinct and array_union #6563 (andygrove)

• test: tag the Azure ABFS suites with AzureTest #6942 (comphead)

• feat: make native Iceberg writes opt-in for 1.2.0 #6962 (andygrove)

• feat: run broadcast joins natively when an unsupported scan is on the build side #6786 (parthchandra)

• feat: expose per-scan S3 HTTP coverage and retries #6429 (sunchao)

• build: use a single jni version across the native workspace #6998 (andygrove)

• feat: add nested benchmark derived from TPC-H #7044 (comphead)

• refactor: simplify shuffle support predicates #7045 (0lai0)

• feat: native RANGE window frames with explicit offset on DATE ORDER BY #4974 (0lai0)

• feat: support PivotFirst aggregate for the optimized PIVOT fast path #4798 (andygrove)

• feat: bump iceberg-rust and write float and double identity partitions natively #6851 (andygrove, mbutrovich)

• refactor: move MergeRows, Explode, aggregate, join and window planning into dedicated modules #7014 (andygrove)

• ci: label Delta pull requests with area:Delta #6965 (dwsmith1983)

• test: write ancient dates with CORRECTED rebase mode in the DATE RANGE window test #7103 (andygrove)

• refactor: share window frame bound offset serialization between lower and upper bounds #7107 (0lai0)

• ci: give the Delta contrib Maven cache its own key prefix #7111 (andygrove)

• chore: fix stddev accumulator size and resolve session libhdfs schemes once per scan rule #7120 (viirya)

• feat: add config to disable or restrict native UDF library loading, and load libraries without a global lock #6775 (andygrove)

• feat: support native wide-decimal hashing #6548 (rich7420)

• refactor: deduplicate split list-array construction #7004 (arminazimi)

• test: verify native timestamp output timezone labels #6930 (LinSimon-901101)

• feat: enable native replace for safe inputs #6680 (sam-1112)

• feat: opt in to join filter pushdown through deterministic residuals #6431 (sunchao)

• chore: limit open PRs for users without write access to 3 #7135 (andygrove)

• test: add window operator benchmarks #7139 (comphead)

• test: cover positional struct reconciliation in native CASE WHEN and COALESCE #7149 (kazuyukitanimura)

• refactor: share Parquet field-id and name resolution between root and nested fields #6990 (andygrove)

• feat: execute scalar Arrow Python UDFs in native Comet #6130 (viirya)

• feat: support AtLeastNNonNulls natively #6180 (rich7420)

• chore: [branch-1.2] change version from 1.2.0-SNAPSHOT to 1.2.0 #7165 (andygrove)

• feat: [branch-1.2] support the S3A magic committer in native Parquet writes (#6940) #7281 (andygrove)

Documentation updates:

• docs: add a simple diagram of how Comet uses executor memory #6237 (andygrove)

• docs: split the tuning guide into multiple pages #6244 (andygrove)

• docs: fix redirects that lead to missing pages #6245 (andygrove)

• docs: add 1.1.0 upgrade notes for fixed, deprecated and removed settings #6248 (andygrove)

• ci: run every tier on release-branch pull requests, and the full suite before an RC #6218 (andygrove)

• docs: remove nonexistent GROUPS frame references #6299 (manuzhang)

• docs: add contributor guide page on timezone handling #6337 (andygrove)

• docs: simplify the memory tuning page #6343 (andygrove)

• docs: correct the Tokio worker default and release_runtime’s shutdown comment #6300 (andygrove)

• docs: add a backporting policy to the contributor guide #6357 (andygrove)

• docs: explain operator plan identity and exchange reuse #6352 (rich7420)

• docs: redraw the home page architecture diagram as a simple SVG #6322 (andygrove)

• docs: update TPC-DS benchmark results for Comet 1.1.0 on Spark 4.2 #6308 (andygrove)

• docs: correct what the Iceberg split-operator plan moves into AQE #6407 (andygrove)

• docs: refresh stale roadmap entries #6309 (andygrove)

• docs: add a Celeborn integration guide #6373 (pingzh)

• docs: request changes for correctness and performance findings in the review skill #6483 (andygrove)

• docs: promote Spark 4.2 from experimental to supported #6417 (andygrove)

• docs: cut the release branch before fixing regressions, and add a diagram #6517 (andygrove)

• docs: explain which Spark operators Comet leaves in place #6551 (andygrove)

• docs: note that the peak native aggregate memory metric is not reported #6557 (andygrove)

• docs: add release-behavior checks and the 1.1.0 audit’s lessons to the review skills #6516 (andygrove)

• docs: note in the 1.1.0 upgrade guide that native Iceberg reads no longer fall back from IRSA #6603 (andygrove)

• docs: name the AQE query stages on the operator support page #6660 (andygrove)

• docs: date the float value and null count divergence to Iceberg 1.9 #6681 (andygrove)

• docs: update the timezone page for the fixes from the timezone EPIC #6686 (andygrove)

• docs: check timezone handling against the contributor guide in the PR review skills #6340 (andygrove)

• docs: add 1.1.0 changelog #6728 (andygrove)

• docs: add an FAQ page with an answer on vectorized Java/Scala UDFs #6778 (andygrove)

• docs: expand the FAQ and refresh the Gluten comparison #6783 (andygrove)

• docs: list the S3A settings the Iceberg native-write gate ignores #6828 (andygrove)

• docs: prefer behavior-named parameters over Spark versions #6776 (andygrove)

• docs: add proposal to establish Apache Comet as a top-level project #6817 (andygrove)

• docs: correct native Iceberg write docs that drifted from the code #7009 (andygrove)

• docs: document casts involving arrays, structs, and maps #6320 (andygrove)

• docs: list known Parquet and Iceberg issues and add a data source audit log #7052 (andygrove)

• docs: consolidate Parquet and Iceberg documentation into one page each #7090 (andygrove)

• docs: [branch-1.2] document known differences new in 1.2.0 in the upgrade guide #7287 (andygrove)

Other:

• chore: add benches remaining scalars #6541 (coderfender)

• chore: add benches for remaining agg functions #6647 (coderfender)

• build(deps): bump rustls from 0.23.44 to 0.23.45 in /native #6701 (dependabot[bot])

• deps: bump to datafusion 55.2.0 #6766 (andygrove)

• build(deps): bump actions/setup-python from 6 to 7 #7046 (dependabot[bot])

Credits

Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor. A PR with commits from more than one person counts for each of them.

   212	Andy Grove
    25	dustin
    21	KUAN-HAO HUANG
    18	Oleks V
    17	Liang-Chi Hsieh
    13	Peter Lee
    11	Chao Sun
    10	LinSimon-901101
     8	ChenChen Lai
     6	Feng Zhang
     6	Unik Dahal
     5	Manu Zhang
     5	Ping Zhang
     5	Yu Hsun Hung
     5	sam-1112
     4	Bhargava Vadlamani
     4	KAZUYUKI TANIMURA
     4	Parth Chandra
     4	Steve Vaughan
     2	Kazantsev Maksim
     2	dependabot[bot]
     2	nat-openai
     1	Alex Huang
     1	Armin Azimi
     1	Bill
     1	DeviousCardi
     1	Erik Bogado
     1	Han-Yin Chang
     1	Jaden Fix
     1	Jeff Wang
     1	Krishna Sudarshan J
     1	Matt Butrovich
     1	Michael Taranov
     1	Po-Han (Stan) Shih · 施博瀚
     1	Samarth Mukhija
     1	YuLun Mao
     1	Yunchi Pang
     1	kid
     1	liupoyi-1031
     1	sandudb

Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.