DataFusion Comet 1.0.0 Changelog#

This release consists of 244 commits from 23 contributors. See credits at the end of this changelog for more information.

Fixed bugs:

  • fix: decline native V1 scans on object_store-unsupported filesystem schemes #4525 (schenksj)

  • fix: exclude release scratch dirs from RAT and license skill docs #4685 (andygrove)

  • fix(shuffle): tolerate non-UTF-8 bytes in get_string (lossy decode) #4524 (schenksj)

  • fix: decline CreateArray with struct-nullability-divergent children #4533 (schenksj)

  • fix: propagate nested cast errors #4675 (manuzhang)

  • fix: reject Parquet INT96 as TimestampNTZ on Spark 3.x #4357 (andygrove)

  • fix: gate str_to_map collations #4701 (manuzhang)

  • fix: repair broken rust-test build on main (init_datasource_exec arg mismatch) #4712 (andygrove)

  • fix: correct stale unix_timestamp NTZ and date_format codegen-default doc text (#4502) #4645 (andygrove)

  • fix: fall back for decimal SUM/AVG over sliding window frames (window audit) #4732 (andygrove)

  • fix: date_trunc schema mismatch and DST handling in non-UTC timezones #4761 (andygrove)

  • fix: size Iceberg delete files in the native scan to avoid dropping deletes #4760 (mbutrovich)

  • fix: gate non-default collations for Spark 4 datetime expressions #4693 (0lai0)

  • fix: ArrayInsert evaluation for null source arrays #4726 (peterxcli)

  • fix: support Spark 4 decimal window avg #4749 (manuzhang)

  • fix: array expression audit follow-ups (#4503) #4713 (andygrove)

  • fix: prevent wrong results from Iceberg native scan exchange reuse with different pushed filters #4812 (mbutrovich)

  • fix: count returning zero when scan is disabled and going through CometSparkColumnarToColumnar #4795 (Dummk0pf)

  • fix: handle null sub-arrays in flatten #4822 (michaelmitchell-bit)

  • fix: correct user guide URL in compatibility fallback messages #4854 (andygrove)

  • fix: report FromUnixTime non-default format as Unsupported #4847 (andygrove)

  • fix: emit JDK libjvm search path from core build script to fix -ljvm CI link failures #4868 (andygrove)

  • fix: register partitioning scalar subqueries for native shuffle to avoid “Subquery N not found” #4869 (mbutrovich)

  • fix: match Spark percentile interpolation precision #4792 (manuzhang)

  • fix: restrict array_filter array_compact fast path to the lambda variable #4848 (andygrove)

  • fix: suppress spurious WriteFilesExec fallback reason on native writes #4928 (andygrove)

  • fix: decode CAST(binary AS string) JVM-compatibly instead of unsafe reinterpret #4763 (andygrove)

  • fix: widen local table scan child nullability to match native kernels #4843 (andygrove)

  • fix: apply 1.11 Iceberg diff changes from #4991 to other versions #5020 (mbutrovich)

  • fix: raise CAST_INVALID_INPUT for invalid calendar dates in ANSI cast to date #5014 (andygrove)

  • fix: materialize ConstantColumnVector on Comet’s serialize/export paths #4532 (schenksj)

  • fix: make native pow implementation compatible with Spark #5033 (andygrove)

  • fix: resolve Comet jar without hardcoding the SNAPSHOT qualifier #5108 (andygrove)

  • fix: honour Spark’s legacy null IN () behavior #5127 (andygrove)

  • fix: honor fail_on_error in native make_decimal #5080 (andygrove)

  • fix: work around DataFusion 54.1.0 Parquet page-index regression #5132 (mbutrovich)

  • fix: throw ARITHMETIC_OVERFLOW for Long.MinValue div -1 under ANSI mode #5084 (andygrove)

  • fix: seed native Parquet scan reader options from session config #5107 (mbutrovich)

  • fix: raise REMAINDER_BY_ZERO for Float/Double under ANSI mode #5081 (andygrove)

  • fix: reduce log verbosity at per-task logging callsites #5155 (mbutrovich)

  • fix: round on Int64 with scale <= -19 now overflows correctly #5082 (andygrove)

  • fix: match Spark’s whitespace trim semantics for casts from string to boolean, integral, float/double and decimal #5150 (andygrove)

  • fix: disambiguate Iceberg scans that share a metadata_location #5180 (mbutrovich)

  • fix: use per-expression eval mode for decimal promotion #5171 (peterxcli)

  • fix: codegen dispatcher null short-circuit swallowed ANSI errors, plus two latent TIME-type gaps #5219 (andygrove)

  • fix: count ReusedSubquery and CometSubqueryBroadcast correctly in extended explain #5206 (andygrove)

Performance related:

  • perf: O(1) PlanDataInjector lookup by op kind #4535 (schenksj)

  • perf: cache full Parquet metadata (incl. page index) via DataFusion’s CachedParquetFileReaderFactory #4707 (mbutrovich)

  • perf: add metadata size hint to Parquet reader to match Iceberg path #4717 (mbutrovich)

  • perf(parquet): revise filter pushdown configuration #4722 (mbutrovich)

  • perf: unwrap identity casts in schema adapter to enable Parquet stats pruning #4730 (mbutrovich)

  • perf: avoid excessive timer calls #4739 (wForget)

  • perf: optimize spark_size in spark-expr #4877 (andygrove)

  • perf: optimize spark_unhex in spark-expr #4876 (andygrove)

  • perf: optimize parse_url (50x speedup) #4893 (andygrove)

  • perf: optimize to_json (2x faster) #4902 (andygrove)

  • perf: optimize spark_cast_int_to_int (100x faster) #4920 (andygrove)

  • perf: optimize try_arithmetic_kernel (up to 4x faster) #4910 (andygrove)

  • perf: optimize spark_arrays_overlap (up to 18x faster) #4906 (andygrove)

  • perf: optimize spark_lpad (up to 2x faster) #4919 (andygrove)

  • perf: optimize CheckOverflow with a shared no-overflow fast path (ANSI and non-ANSI) #4937 (andygrove)

  • perf: skip the null-masking pass in DecimalRescaleCheckOverflow when nothing overflows #4938 (andygrove)

  • perf: vectorize floating-point-to-decimal cast #4940 (andygrove)

  • perf: optimize spark_get_json_object (4x faster) #4907 (andygrove)

  • perf: optimize spark_regexp_extract #4894 (andygrove)

  • perf: vectorize spark_unscaled_value (9x faster) #4972 (u70b3)

  • perf: optimize cast_binary_to_string binary-format styles (up to 27x faster) #4912 (andygrove)

  • perf: optimize parse_string_to_decimal (30-40% faster) #4916 (andygrove)

  • perf: optimize date_parser / cast string to date (up to 2x faster) #4917 (andygrove)

  • perf: optimize cast_decimal128_to_utf8 in datafusion-comet-spark-expr #4924 (andygrove)

  • perf: optimize date_trunc (>2x faster) #4915 (andygrove)

  • perf: optimize spark_cast_float64_to_utf8 (~40% faster) #4918 (andygrove)

  • perf: dedupe Iceberg residuals and delete files in native scan serde #4982 (mbutrovich)

  • perf: optimize spark_ceil (3x faster) #4926 (andygrove)

  • perf: bypass shuffle BatchCoalescer for already-sized batches #5003 (andygrove)

  • perf: encode shuffle IPC schema once per writer instead of per block #5006 (andygrove)

  • refactor: drop redundant concat layer in single-partition shuffle #5004 (andygrove)

  • fix: make native cast from float/double to decimal compatible with Spark #5136 (andygrove)

  • perf: intern QueryContext SQL text into a per-plan pool (up to 20x smaller serialized plans for TPC-DS) #5204 (andygrove)

  • perf: avoid rebuilding untouched operators in PlanDataInjector.injectPlanData #5220 (andygrove)

Implemented enhancements:

  • feat: surface native parquet read failures as FAILED_READ_FILE #4536 (schenksj)

  • feat: opt concat into codegen dispatch for non-UTF8_BINARY collations #4640 (andygrove)

  • feat: opt sort_array into codegen dispatch under strict floating-point mode #4637 (andygrove)

  • feat: support MapType input for ElementAt #4697 (0lai0)

  • feat: extend native windows support #4209 (comphead)

  • feat: add array_prepend support #4716 (andygrove)

  • feat: surface native opt-in expressions as compatible-by-default with a COMET-INFO plan hint #4721 (andygrove)

  • feat: support StringSplitSQL for split_part #4592 (michaelmitchell-bit)

  • feat: support exact percentile and median aggregates natively #4542 (andygrove)

  • feat: support TimestampNTZ inputs natively for hour/minute/second #4753 (andygrove)

  • feat: add unsupported metadata column names to fallback reasons #4758 (hsiang-c)

  • feat: Add support for base64 #4778 (andygrove)

  • feat: support interval types and make_ym_interval / make_dt_interval #4541 (andygrove)

  • feat: support PreciseTimestampConversion for native batch time-window grouping #4784 (andygrove)

  • feat: support Azure authentication for native Parquet scan #4783 (Dummk0pf)

  • feat: support shuffle array expression #4797 (andygrove)

  • feat: rebalance associative bitwise/Add/Multiply chains to avoid protobuf recursion limit #4588 (schenksj)

  • feat: support grouping() and grouping_id() indicator functions #4815 (andygrove)

  • feat: Stage based fallback #4519 (karuppayya)

  • feat: release tokio runtime on driver/executor exit #4734 (wForget)

  • feat: route Unsupported through codegen dispatch for opt-in serdes #4728 (andygrove)

  • docs: comet docs design overhaul- phase 1 #4353 (pranamya123)

  • feat: name incompatible aggregate functions in mixed-execution fallback reason #4750 (andygrove)

  • feat: implement native empty2null spark inner function #4683 (kazantsev-maksim)

  • feat: enable mixed partial/final execution for sum and non-decimal avg #4861 (andygrove)

  • feat: Add experimental support for accelerated PyArrow UDFs #4234 (andygrove)

  • feat: support Iceberg 1.11, audit existing Iceberg diffs, bump iceberg-rust dep, add run-iceberg-tests CI trigger #4840 (mbutrovich)

  • feat: support approx_percentile / percentile_approx aggregate #4801 (andygrove)

  • feat: remove constraint on array of nested elements #4714 (hsiang-c)

  • feat: core SPI for contrib leaf scans (CometScanWithPlanData) [Delta contrib split, part 1] #4700 (schenksj)

  • feat: support gzip compression in native Parquet writes #4930 (andygrove)

  • feat: support size() for MapType input #4580 (marvelshan)

  • feat: hint at native pyarrow UDF path when the feature is disabled #4892 (andygrove)

  • feat: Implement TimeType support - Infrastructure - shuffle #4398 (YutaLin)

  • feat: fall back to Spark for collated predicate operands #4948 (comphead)

  • feat: add withAlternative alias mechanism for CometConf #4979 (andygrove)

  • feat: add spark.comet.shuffle.maxBufferBytes to cap native shuffle writer memory #4989 (andygrove)

  • feat: support approx_count_distinct aggregate expression #4819 (andygrove)

  • feat: Iceberg table format V3: native table decryption, fall back for other V3 features #4991 (mbutrovich)

  • feat: support multiply_dt_interval with codegen dispatch #4900 (peterxcli)

  • feat: Support Spark levenshtein expression in native execution #4105 (Myx778)

  • feat: native collect_list / array_agg aggregate #4720 (andygrove)

  • feat: add CalendarIntervalType support #4898 (peterxcli)

  • feat: native randstr implementation compatible with Spark #5035 (andygrove)

  • feat: support interval codegen dispatch for nested values and native shuffle #4976 (peterxcli)

  • feat: add codegen dispatch fallback for CometCast incompatible/unsupported cases (including legacy config paths) #5079 (comphead)

  • feat: support SampleExec natively for sampling without replacement #5110 (andygrove)

  • feat: disable native columnar-to-row conversion by default #5114 (andygrove)

  • feat: expose Comet version as spark.comet.version runtime config #5049 (andygrove)

  • feat: native uuid() implementation compatible with Spark #5034 (andygrove)

  • feat: support Iceberg metadata columns _pos, _spec, _file, and _partition #4752 (parthchandra)

  • feat: report native vs codegen-dispatch expression coverage in extended explain #5201 (andygrove)

Documentation updates:

  • docs: update release_process for changelog #4668 (mbutrovich)

  • docs: restore 0.13 user guide dropped in 0.17 release #4704 (andygrove)

  • docs: fix prettier check error #4748 (wForget)

  • docs: refresh expression audit notes for resolved correctness issues #4762 (andygrove)

  • docs: mark try_avg and try_sum as natively supported #4776 (andygrove)

  • docs: update Iceberg docs to clarify S3 storage credentials #4767 (mbutrovich)

  • docs: render captioned sidebar sections for versioned user guides #4699 (andygrove)

  • docs: triage bug vs enhancement and apply type labels consistently #4768 (andygrove)

  • docs: mark make_dt_interval and make_ym_interval as supported #4790 (andygrove)

  • docs: correct window function support status and limitations #4833 (andygrove)

  • docs: minor docs site improvements #4855 (andygrove)

  • docs: remove references to closed and fixed issues from compatibility guide #4856 (andygrove)

  • docs: fix sidebar drawer mis-tap, missing aria-expanded, homepage permalink, and ASF links page #4858 (pranamya123)

  • docs: announce JDK 11 and Spark 3.4 deprecation for 1.1.0 removal #4857 (andygrove)

  • docs: note invalid UTF-8 scan limitation and FFI import risk #4846 (andygrove)

  • docs: update Gluten comparison with AWS Labs benchmark and Comet strengths #4873 (andygrove)

  • docs: add scalar expression optimization guide, skill, and performance audits #4933 (andygrove)

  • doc: Document scan tuning for spark.sql.files.maxPartitionBytes #4931 (comphead)

  • docs: add 0.17.1 changelog #4961 (mbutrovich)

  • docs: document Spark version adoption and support-lifetime policy #4977 (andygrove)

  • docs: show implementation kind for each expression #5028 (andygrove)

  • docs: add blog posts and talks page #5043 (andygrove)

  • docs: avoid “not a blocker” comments in reviews for review skill #5057 (mbutrovich)

  • docs: refresh stale issue references and normalize issue link format #5062 (andygrove)

  • docs: fix references to configuration keys that do not exist #5063 (andygrove)

  • docs: define post-1.0 versioning policy and legacy config process #5056 (andygrove)

  • docs: expand tuning guide with performance and memory configs #4908 (andygrove)

  • docs: document known correctness issues in the compatibility guide #5085 (andygrove)

  • docs: update post 1.0.0 roadmap #5064 (mbutrovich)

  • docs: drop compatibility notes for bugs that are now fixed #5154 (andygrove)

  • docs: document ReusedExchange caveat and both CometSparkToColumnar names in operator-count exclusions #5240 (andygrove)

Other:

  • chore: fix generate-release-docs.sh for per-Spark-version doc layout #4662 (mbutrovich)

  • chore: add branch protection to release branches, update release_process.md #4665 (mbutrovich)

  • chore: start 0.18.0 development #4664 (mbutrovich)

  • test: cover nested complex casts #4608 (manuzhang)

  • refactor: move string expression support checks to getSupportLevel #4676 (andygrove)

  • refactor: move arithmetic and math support checks to getSupportLevel #4674 (andygrove)

  • refactor: move aggregate expression support checks to getSupportLevel #4678 (andygrove)

  • refactor: move array expression support checks to getSupportLevel #4677 (andygrove)

  • chore: add array tests with NaN handling #4686 (comphead)

  • chore: tweak CI execution memory params #4687 (comphead)

  • chore: add ordering tests for array_union #4688 (comphead)

  • chore: fix ConstantFolding rule exclusion for benchmarks #4689 (comphead)

  • chore(deps): bump actions/checkout from 6 to 7 #4690 (dependabot[bot])

  • chore(deps): bump the all-other-cargo-deps group in /native with 2 updates #4691 (dependabot[bot])

  • chore(deps): bump itertools from 0.14.0 to 0.15.0 in /native #4692 (dependabot[bot])

  • chore: update documentation links for 0.17.0 release #4698 (andygrove)

  • chore: add optional CI flow for parquet writes #4696 (comphead)

  • deps: Upgrade to DataFusion 54.0.0 #4062 (andygrove)

  • chore: apply same SBT profile for Spark 4.0 #4709 (comphead)

  • refactor: remove map_contains_key serde #4703 (manuzhang)

  • chore: Spark 4.0 restore DEDICATED_JVM_SBT_TESTS #4711 (comphead)

  • refactor: share Spark 4.1+ shim sources to remove duplication #4710 (andygrove)

  • chore(deps): bump object_store from 0.13.2 to 0.14.0 in /native #4737 (dependabot[bot])

  • chore(deps): bump the all-other-cargo-deps group in /native with 3 updates #4736 (dependabot[bot])

  • chore(deps): bump actions/cache from 5 to 6 #4735 (dependabot[bot])

  • deps: revert object_store to 0.13.2 #4740 (mbutrovich)

  • chore: fix 4.0.2 diff to handle CometWindowsExec properly #4743 (comphead)

  • chore: surface DataFusion 54 PruningMetrics and Ratio in CometNativeScan metrics #4733 (mbutrovich)

  • refactor: gate CometUnaryMinus input types in getSupportLevel #4759 (andygrove)

  • test: enable literal sha2 now that native engine handles scalar args #4773 (andygrove)

  • ci: republish docs site when generated-doc sources change #4777 (andygrove)

  • test: enable SparkSessionExtensionSuite with Comet #4772 (andygrove)

  • chore: start 1.0.0 development #4794 (andygrove)

  • chore(deps): bump rand from 0.10.1 to 0.10.2 in /native in the all-other-cargo-deps group #4805 (dependabot[bot])

  • deps: bump iceberg-rust version to a rev with ArrowReader fix #4826 (sandugood)

  • chore: Use native impl of soundex function #4824 (kazantsev-maksim)

  • refactor(shuffle): Introduce PartitionWriter interface to decouple shuffle partitioning from storage #4779 (wForget)

  • chore: fallback for spark.sql.mapKeyDedupPolicy == LAST_WIN #4863 (comphead)

  • chore: remove hdfs Comet crate #4904 (comphead)

  • test: cover invalid unhex inputs in SQL file tests #4890 (andygrove)

  • chore: Codegen fallback explain for plan #4891 (comphead)

  • test: add version check on an Iceberg test for Variant type #4935 (mbutrovich)

  • chore: use Datafusion substring #4161 (comphead)

  • chore: Support native writer Overwrite and ErrorIfExists modes #4946 (comphead)

  • refactor: move AttributeReference and KnownFloatingPointNormalized expression support checks to getSupportLevel #4745 (peterxcli)

  • chore(deps): bump actions/setup-node from 6 to 7 #4955 (dependabot[bot])

  • chore: bump spark-4.2 profile to the released 4.2.0 #4960 (andygrove)

  • chore(deps): bump the all-other-cargo-deps group across 1 directory with 6 updates #4956 (dependabot[bot])

  • chore(deps): bump lz4_flex from 0.13.1 to 0.14.0 in /native #4958 (dependabot[bot])

  • chore: codeql dependabot fix #4962 (comphead)

  • chore(deps): bump the codeql-actions group with 2 updates #4970 (dependabot[bot])

  • test: promote try_to_date/try_to_timestamp SQL tests to native coverage #4973 (u70b3)

  • chore: remove dead Parquet parallel-IO configs #4981 (andygrove)

  • refactor: remove untested async columnar shuffle #4985 (andygrove)

  • deps: bump to datafusion 54.1.0 and latest iceberg-rust #4996 (mbutrovich)

  • chore: use DF array_repeat and array_compact #4741 (comphead)

  • chore: remove legacy use.lazyMaterialization #4998 (kazuyukitanimura)

  • chore(deps): bump actions/stale from 10.3.0 to 10.4.0 #4879 (dependabot[bot])

  • chore(deps): bump base64 from 0.22.1 to 0.23.0 in /native #5018 (dependabot[bot])

  • chore(deps): bump actions/setup-python from 6 to 7 #5017 (dependabot[bot])

  • chore(deps): bump the codeql-actions group with 2 updates #5016 (dependabot[bot])

  • chore(deps): bump actions/cache from 5 to 6 #4882 (dependabot[bot])

  • chore(deps): bump actions/checkout from 6 to 7 #4881 (dependabot[bot])

  • refactor: group orphan spark.comet.explain.* configs under one prefix #5026 (andygrove)

  • refactor: unify shuffle configs under spark.comet.shuffle.* prefix #4986 (andygrove)

  • test: add isolated and small-batch benchmarks for columnar-to-row conversion #5113 (andygrove)

  • chore: stop enabling incompatible casts in plan stability suite #5139 (andygrove)

  • chore: replace deprecated symbol literals with $”col” in tests #5168 (andygrove)

  • test: cover ArrayExists three-valued logic config #5000 (manuzhang)

  • chore(deps): bump actions/stale from 10.4.0 to 11.0.0 #5163 (dependabot[bot])

  • chore: remove Array -> Seq round trips flagged by the 2.13 copy deprecation #5170 (andygrove)

  • chore: supply explicit empty argument lists for 2.13 #5175 (andygrove)

  • chore: clean up Scala compiler warnings #5141 (andygrove)

  • chore: drop deprecated .toIterable from protobuf builder calls #5173 (andygrove)

  • chore: bump Spark 3.5 to 3.5.9 #5181 (andygrove)

  • chore: bump Spark 4.1 to 4.1.3 #5183 (andygrove)

  • refactor: use arrow make_comparator for nested structural equality in arrays_overlap and array_position #5176 (peterxcli)

  • chore: bump Spark 4.0 to 4.0.4 #5182 (andygrove)

  • chore: remove unused spark.comet.exceptionOnDatetimeRebase config #5221 (andygrove)

  • refactor: rename pyarrowUdf config to pyarrowUDF and fix stale config docs #5197 (andygrove)

  • chore: cargo update for 1.0 release #5226 (mbutrovich)

Credits#

Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.

   132	Andy Grove
    28	Matt Butrovich
    19	Oleks V
    18	dependabot[bot]
     8	Scott Schenkein
     7	Manu Zhang
     7	Peter Lee
     4	Zhen Wang
     2	ChenChen Lai
     2	Kazantsev Maksim
     2	Prames Maanikam
     2	Pranamya Vadlamani
     2	hsiang-c
     2	kid
     1	Bolin Lin
     1	KAZUYUKI TANIMURA
     1	Karuppayya
     1	Mitchell
     1	Myx778
     1	Parth Chandra
     1	Zaki
     1	alexander domenti
     1	michaelmitchell-bit

Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.

On this page