<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->

# Apache DataFusion Ballista 55.0.0 Changelog

This release consists of 322 commits from 25 contributors. See credits at the end of this changelog for more information.

**Fixed bugs:**

- fix(scheduler): persist terminal job state before cache eviction [#2037](https://github.com/apache/datafusion-ballista/pull/2037) (phillipleblanc)
- fix: spill sort shuffle writer when the memory pool rejects a reservation grow [#2061](https://github.com/apache/datafusion-ballista/pull/2061) (andygrove)
- fix: enable the executor client pool by default [#2069](https://github.com/apache/datafusion-ballista/pull/2069) (andygrove)
- fix: map a task's partition through UnionExec when restricting file scans [#2070](https://github.com/apache/datafusion-ballista/pull/2070) (andygrove)
- fix: keep EmptyExec partition count intact across stage serialization [AQE] [#2062](https://github.com/apache/datafusion-ballista/pull/2062) (andygrove)
- fix: remove the static planner SortMergeJoinExec broadcast conversion [#2072](https://github.com/apache/datafusion-ballista/pull/2072) (andygrove)
- fix: propagate sort shuffle memory_limit_per_task_bytes to executors [#2090](https://github.com/apache/datafusion-ballista/pull/2090) (andygrove)
- fix: executor hanging on ctrl+c [#2107](https://github.com/apache/datafusion-ballista/pull/2107) (sandugood)
- fix: Various fixes for AQE join selection [#2124](https://github.com/apache/datafusion-ballista/pull/2124) (andygrove)
- fix: Enable more TPC-DS queries [#2064](https://github.com/apache/datafusion-ballista/pull/2064) (andygrove)
- fix(executor): keep heartbeating while all task slots are busy [#2159](https://github.com/apache/datafusion-ballista/pull/2159) (andygrove)
- fix: unskip five TPC-DS queries by rewriting generator column names [#2183](https://github.com/apache/datafusion-ballista/pull/2183) (andygrove)
- fix: stop every task running every union branch at low partition counts [#2186](https://github.com/apache/datafusion-ballista/pull/2186) (andygrove)
- fix(scheduler): execute null-aware anti joins in one task [#2188](https://github.com/apache/datafusion-ballista/pull/2188) (phillipleblanc)
- fix(scheduler): keep GROUP BY-less aggregates when propagating empty stages [#2194](https://github.com/apache/datafusion-ballista/pull/2194) (andygrove)
- fix(scheduler): harden PropagateEmptyExecRule against grouping sets, embedded projections, and dropped exchange partitioning [#2200](https://github.com/apache/datafusion-ballista/pull/2200) (andygrove)
- fix(cli): sort jobs by stage completion ratio [#1947](https://github.com/apache/datafusion-ballista/pull/1947) (QuakeWang)
- fix(scheduler): preserve stages on transition errors [#2215](https://github.com/apache/datafusion-ballista/pull/2215) (QuakeWang)
- fix(scheduler): fail running jobs when all executors are lost [#2212](https://github.com/apache/datafusion-ballista/pull/2212) (akshaychitneni)
- fix(executor): bind the gRPC listener before registering with the scheduler [#2225](https://github.com/apache/datafusion-ballista/pull/2225) (andygrove)
- fix: update mTLS example for poll_loop arguments [#2229](https://github.com/apache/datafusion-ballista/pull/2229) (phillipleblanc)
- fix(scheduler): route launch-failure executor removal through ExecutorLost [#2240](https://github.com/apache/datafusion-ballista/pull/2240) (andygrove)
- fix: classify wrapped FetchFailed as fetch partition error [#2202](https://github.com/apache/datafusion-ballista/pull/2202) (villebro)
- fix(core): fix main build errors [#2254](https://github.com/apache/datafusion-ballista/pull/2254) (avantgardnerio)
- fix: pin datafusion versions to 54.0.0 for python subproject [#2277](https://github.com/apache/datafusion-ballista/pull/2277) (milenkovicm)
- fix(scheduler): key AQE shuffle output by job_id [#2278](https://github.com/apache/datafusion-ballista/pull/2278) (villebro)
- fix(scheduler): default no-executors grace to executor timeout (>= heartbeat) [#2279](https://github.com/apache/datafusion-ballista/pull/2279) (akshaychitneni)
- fix(docs): k8s executors dial the registration Service; scheduler needs maxUnavailable: 1 [#2289](https://github.com/apache/datafusion-ballista/pull/2289) (avantgardnerio)
- fix(ci): apply cargo fmt to chaos-executor [#2304](https://github.com/apache/datafusion-ballista/pull/2304) (andygrove)
- fix(scheduler): label every operator in the DOT graph [#2305](https://github.com/apache/datafusion-ballista/pull/2305) (andygrove)
- fix(core): report the real partitioning from ShuffleWriterExec [#2307](https://github.com/apache/datafusion-ballista/pull/2307) (andygrove)
- fix(scheduler): drop an executor's cached gRPC client when it is removed [#2310](https://github.com/apache/datafusion-ballista/pull/2310) (Dandandan)
- fix: repair AQE plan rewrites, remove a redundant shuffle, and enable adaptive query planning by default [#2315](https://github.com/apache/datafusion-ballista/pull/2315) (Dandandan)
- fix(tui): use the shared API wire types instead of local copies [#2258](https://github.com/apache/datafusion-ballista/pull/2258) (andygrove)
- fix(chaos): harden k8s backend — supervise port-forward, guard fixture dir [#2346](https://github.com/apache/datafusion-ballista/pull/2346) (akshaychitneni)
- fix(docker): base runtime images on ubuntu:26.04 [#2380](https://github.com/apache/datafusion-ballista/pull/2380) (avantgardnerio)
- fix(scheduler): fix test job-completion polling [#2354](https://github.com/apache/datafusion-ballista/pull/2354) (NoahKusaba)
- fix(ci): pin tinyvec in the MSRV dependency checker [#2421](https://github.com/apache/datafusion-ballista/pull/2421) (andygrove)
- fix(executor): wake graceful shutdown after tasks drain [#2393](https://github.com/apache/datafusion-ballista/pull/2393) (1fanwang)
- fix(scheduler): repair test broken by a stale-base merge [#2439](https://github.com/apache/datafusion-ballista/pull/2439) (NoahKusaba)
- fix: derive stage input/output rows from the plan's leaves and root instead of summing every operator [#2435](https://github.com/apache/datafusion-ballista/pull/2435) (Dandandan)
- fix(ci): pull the MinIO test image from quay.io [#2447](https://github.com/apache/datafusion-ballista/pull/2447) (comphead)
- fix(ci): resolve datafusion.proto from datafusion-proto-models [#2390](https://github.com/apache/datafusion-ballista/pull/2390) (AnuragRaut08)
- fix(ci): upload both cluster logs from the container benchmark jobs [#2460](https://github.com/apache/datafusion-ballista/pull/2460) (amitvijapur)
- fix(ci): run the S3 integration tests against RustFS instead of MinIO [#2488](https://github.com/apache/datafusion-ballista/pull/2488) (NoahKusaba)
- fix(executor): forward arrow-ipc-optimizations to ballista-core [#2373](https://github.com/apache/datafusion-ballista/pull/2373) (stag7824)
- fix(scheduler): refund the vcores of running tasks when a job is aborted [#2477](https://github.com/apache/datafusion-ballista/pull/2477) (amitvijapur)
- fix(scheduler): fail jobs whose tasks cannot be prepared [#2492](https://github.com/apache/datafusion-ballista/pull/2492) (andygrove)
- fix(scheduler): declare distributed EXPLAIN output columns not-null [#2494](https://github.com/apache/datafusion-ballista/pull/2494) (andygrove)
- fix(core): refuse queries that mix information_schema with other tables [#2493](https://github.com/apache/datafusion-ballista/pull/2493) (andygrove)
- fix(release): prepare the 55 release process [#2527](https://github.com/apache/datafusion-ballista/pull/2527) (andygrove)
- fix(docker): publish images from release tag pushes [#2526](https://github.com/apache/datafusion-ballista/pull/2526) (andygrove)

**Performance related:**

- perf(kll): batch-oriented ingest KLL [#2216](https://github.com/apache/datafusion-ballista/pull/2216) (avantgardnerio)
- perf(core): zero-copy block receive on the remote fetch path [#2312](https://github.com/apache/datafusion-ballista/pull/2312) (Dandandan)
- perf(core): push a consumer's row limit into RangeShuffleReaderExec [#2334](https://github.com/apache/datafusion-ballista/pull/2334) (Dandandan)
- perf(scheduler): push filters once, not on every replan [#2344](https://github.com/apache/datafusion-ballista/pull/2344) (Dandandan)
- perf(scheduler): only give a broadcast build side its own stage when the probe is partitioned [#2342](https://github.com/apache/datafusion-ballista/pull/2342) (Dandandan)
- perf(scheduler): read an unfiltered small scan inline instead of staging it [#2338](https://github.com/apache/datafusion-ballista/pull/2338) (Dandandan)
- perf(core): write one sort-shuffle file per task instead of one per input [#2316](https://github.com/apache/datafusion-ballista/pull/2316) (Dandandan)
- perf(core): reuse shuffle partition-index buffers across batches [#2359](https://github.com/apache/datafusion-ballista/pull/2359) (NoahKusaba)
- perf(core): strength reduce hash partition modulo [#2363](https://github.com/apache/datafusion-ballista/pull/2363) (NoahKusaba)
- perf(core): skip IPC validation on the sort-shuffle read paths [#2365](https://github.com/apache/datafusion-ballista/pull/2365) (NoahKusaba)
- perf(scheduler): stop re-encoding the session config for every task [#2358](https://github.com/apache/datafusion-ballista/pull/2358) (NoahKusaba)
- perf(core): remove the inert shuffle-read sort [#2414](https://github.com/apache/datafusion-ballista/pull/2414) (NoahKusaba)
- perf(scheduler): share the file statistics cache across sessions [#2498](https://github.com/apache/datafusion-ballista/pull/2498) (andygrove)

**Implemented enhancements:**

- feat: make compression codec configurable [#2039](https://github.com/apache/datafusion-ballista/pull/2039) (sandugood)
- feat: make broadcast_join_threshold_bytes/rows authoritative under AQE and the static planner [#2086](https://github.com/apache/datafusion-ballista/pull/2086) (andygrove)
- feat: add BALLISTA_PROTOCOL_VERSION + k8s health probes [#2088](https://github.com/apache/datafusion-ballista/pull/2088) (avantgardnerio)
- feat: allow an external semaphore for executor task concurrency [#1892](https://github.com/apache/datafusion-ballista/pull/1892) (Jeadie)
- feat: log sort shuffle spills at WARN, keep write completion at DEBUG [#2091](https://github.com/apache/datafusion-ballista/pull/2091) (andygrove)
- feat(benchmarks): persist TPC-H results resiliently and add a compare subcommand [#2102](https://github.com/apache/datafusion-ballista/pull/2102) (andygrove)
- feat(core): RuntimeStatsExec — passthrough tap for row counts + quantile sketches [#2094](https://github.com/apache/datafusion-ballista/pull/2094) (avantgardnerio)
- feat(core): BufferExec — generic flow-control op with a memory-pool Dam mode [#2095](https://github.com/apache/datafusion-ballista/pull/2095) (avantgardnerio)
- feat: Improve TPC-H performance with AQE on [#2084](https://github.com/apache/datafusion-ballista/pull/2084) (andygrove)
- feat(benchmarks): add --skip to omit queries from a tpch suite run [#2129](https://github.com/apache/datafusion-ballista/pull/2129) (andygrove)
- feat(core): report the actual sort shuffle spill trigger in logs [#2143](https://github.com/apache/datafusion-ballista/pull/2143) (andygrove)
- feat(core): add UnorderedRangeRepartitionExec — value-range router over unordered inputs [#2123](https://github.com/apache/datafusion-ballista/pull/2123) (avantgardnerio)
- feat(executor): default to a bounded, auto-sized memory pool [#2160](https://github.com/apache/datafusion-ballista/pull/2160) (andygrove)
- feat(core): add OrderedRangeRepartitionExec — N sorted overlapping → K sorted disjoint [#2169](https://github.com/apache/datafusion-ballista/pull/2169) (avantgardnerio)
- feat(core): add PerPartitionFilterExec — per-input-partition boolean predicate [#2195](https://github.com/apache/datafusion-ballista/pull/2195) (avantgardnerio)
- feat(core): rewrite NOT IN subqueries into distributable anti join plans [#2199](https://github.com/apache/datafusion-ballista/pull/2199) (andygrove)
- feat(scheduler): range-repartition rule primitives (batteries-included) [#2196](https://github.com/apache/datafusion-ballista/pull/2196) (avantgardnerio)
- feat: make gRPC max message size consistently configurable [#2173](https://github.com/apache/datafusion-ballista/pull/2173) (andygrove)
- feat: add `poll_now_notify` to `poll_loop` and `on_work_available` callback [#1893](https://github.com/apache/datafusion-ballista/pull/1893) (Jeadie)
- feat(scheduler): expose existing job state event stream [#1891](https://github.com/apache/datafusion-ballista/pull/1891) (Jeadie)
- feat(scheduler): Redirect '/' to nighlies.a.o WebTUI [#2059](https://github.com/apache/datafusion-ballista/pull/2059) (martin-g)
- feat(core): metrics() for OrderedRangeRepartitionExec and RuntimeStatsExec [#2250](https://github.com/apache/datafusion-ballista/pull/2250) (avantgardnerio)
- feat: add DistributedQueryExec::explain_executed_plan and show logical plan in display [#2042](https://github.com/apache/datafusion-ballista/pull/2042) (andygrove)
- feat(core): PBWAG — parallel BWAG wrapper [#2253](https://github.com/apache/datafusion-ballista/pull/2253) (avantgardnerio)
- feat(core): RangeShuffleReaderExec — ordering-preserving shuffle reader (purely additive) [#2255](https://github.com/apache/datafusion-ballista/pull/2255) (avantgardnerio)
- feat(core): RangeFilterExec — sorted-input per-partition range filter (purely additive) [#2262](https://github.com/apache/datafusion-ballista/pull/2262) (avantgardnerio)
- feat(history): add the event-log schema, writer, and reader [#2260](https://github.com/apache/datafusion-ballista/pull/2260) (andygrove)
- feat(scheduler): record a per-job event log behind --event-log-dir [#2264](https://github.com/apache/datafusion-ballista/pull/2264) (andygrove)
- feat: pluggable job id generator [#2269](https://github.com/apache/datafusion-ballista/pull/2269) (milenkovicm)
- feat: parallel BWAG for the range-window shape (h2o Q8) [#2223](https://github.com/apache/datafusion-ballista/pull/2223) (avantgardnerio)
- feat: embedded flight proxy explicit config entry [#2288](https://github.com/apache/datafusion-ballista/pull/2288) (sandugood)
- feat(core): quantile sketching for any fixed-width ORDER BY key [#2294](https://github.com/apache/datafusion-ballista/pull/2294) (avantgardnerio)
- feat(core): Replace TDigest with KLL [#2337](https://github.com/apache/datafusion-ballista/pull/2337) (avantgardnerio)
- feat: parallel prefix scan for UNBOUNDED PRECEDING window aggregates [#2211](https://github.com/apache/datafusion-ballista/pull/2211) (avantgardnerio)
- feat: remove serialisation of built-in file formats from ballista [#2348](https://github.com/apache/datafusion-ballista/pull/2348) (milenkovicm)
- feat(core): range shuffle reads only the bytes a consumer's range covers [#2364](https://github.com/apache/datafusion-ballista/pull/2364) (avantgardnerio)
- feat(scheduler): add the history server [#2265](https://github.com/apache/datafusion-ballista/pull/2265) (andygrove)
- feat: scheduler REST & GRPC returns ballista version headers [#2378](https://github.com/apache/datafusion-ballista/pull/2378) (milenkovicm)
- feat: cleanup & simplify ballista config [#2387](https://github.com/apache/datafusion-ballista/pull/2387) (milenkovicm)
- feat(history): mark running event logs with a .running suffix until job completion [#2405](https://github.com/apache/datafusion-ballista/pull/2405) (milenkovicm)
- feat(scheduler): stage a join's build side before deciding the join [#2434](https://github.com/apache/datafusion-ballista/pull/2434) (andygrove)
- feat(scheduler): document REST API with OpenAPI and serve /api/openapi.json [#2397](https://github.com/apache/datafusion-ballista/pull/2397) (DebadityaHait)
- feat(scheduler): record scheduler id in job starts [#2462](https://github.com/apache/datafusion-ballista/pull/2462) (villebro)
- feat(scheduler): add a setting to disable the shared file statistics cache [#2510](https://github.com/apache/datafusion-ballista/pull/2510) (andygrove)
- feat: Column statistics support v1 [#1916](https://github.com/apache/datafusion-ballista/pull/1916) (coderfender)
- feat: error when the client and scheduler have different major versions [#2514](https://github.com/apache/datafusion-ballista/pull/2514) (andygrove)
- feat: Arrow Flight SQL frontend for the scheduler + ADBC support [#2416](https://github.com/apache/datafusion-ballista/pull/2416) (andygrove)

**Documentation updates:**

- docs: add user personas as a review contract for additive changes [#2002](https://github.com/apache/datafusion-ballista/pull/2002) (andygrove)
- docs: include TestPyPI wheels link in release vote email [#2010](https://github.com/apache/datafusion-ballista/pull/2010) (andygrove)
- docs: remove outdated TPC-H benchmark results from READMEs [#2015](https://github.com/apache/datafusion-ballista/pull/2015) (andygrove)
- Rename `task_slots` -> `vcores` (executor spec + CLI) [#2045](https://github.com/apache/datafusion-ballista/pull/2045) (avantgardnerio)
- docs: add benchmarking page to the contributors guide [#2075](https://github.com/apache/datafusion-ballista/pull/2075) (andygrove)
- Multi-partition tasks: partition_slice plumbing + SortShuffleWriter refactor [#2038](https://github.com/apache/datafusion-ballista/pull/2038) (avantgardnerio)
- docs(config): rewrite max_partitions_per_task description to be timeless [#2093](https://github.com/apache/datafusion-ballista/pull/2093) (avantgardnerio)
- docs: refresh SF1000 benchmark results on 2x16-core AQE-on reference cluster [#2109](https://github.com/apache/datafusion-ballista/pull/2109) (andygrove)
- Remove the hash-partitioning shuffle writer [#2106](https://github.com/apache/datafusion-ballista/pull/2106) (andygrove)
- docs: correct scan partitioning description in tuning guide [#2125](https://github.com/apache/datafusion-ballista/pull/2125) (andygrove)
- Generate the config reference documentation from the config registry [#2146](https://github.com/apache/datafusion-ballista/pull/2146) (andygrove)
- docs: refresh benchmarking.md with current SF1000 results (Ballista main @ 696ca29b, Spark 3.4 baseline) [#2161](https://github.com/apache/datafusion-ballista/pull/2161) (andygrove)
- docs: refresh benchmarking.md with dedicated shuffle PVC results [#2171](https://github.com/apache/datafusion-ballista/pull/2171) (andygrove)
- docs: add 54.0.0 changelog [#2012](https://github.com/apache/datafusion-ballista/pull/2012) (andygrove)
- feat(core,scheduler): transport RuntimeStatsExec reports to scheduler; log merged quantile cuts per stage [#2175](https://github.com/apache/datafusion-ballista/pull/2175) (avantgardnerio)
- docs(kubernetes): two-Service pattern for scheduler bootstrap [#2245](https://github.com/apache/datafusion-ballista/pull/2245) (avantgardnerio)
- docs: add 54.1.0 changelog to main [#2271](https://github.com/apache/datafusion-ballista/pull/2271) (andygrove)
- docs: document Web TUI [#2268](https://github.com/apache/datafusion-ballista/pull/2268) (jvsilva12600009)
- docs: clean up stale and broken documentation links [#2287](https://github.com/apache/datafusion-ballista/pull/2287) (AntiO2)
- docs: document DataFusion dependency cycle in contributor guide [#2293](https://github.com/apache/datafusion-ballista/pull/2293) (andygrove)
- docs: explain why Ballista uses a blocking shuffle [#2308](https://github.com/apache/datafusion-ballista/pull/2308) (andygrove)
- docs(bench): refresh TPC-H SF1000 results at 67b3a19b [#2417](https://github.com/apache/datafusion-ballista/pull/2417) (andygrove)
- docs: move developer docs into the published contributors guide [#2481](https://github.com/apache/datafusion-ballista/pull/2481) (NoahKusaba)
- docs: say vcores instead of task slots and concurrency levels [#2482](https://github.com/apache/datafusion-ballista/pull/2482) (NoahKusaba)
- docs: include examples and config tables from source instead of copying them [#2480](https://github.com/apache/datafusion-ballista/pull/2480) (NoahKusaba)
- docs: describe how joins are chosen under AQE [#2479](https://github.com/apache/datafusion-ballista/pull/2479) (NoahKusaba)
- ci: run ruff over the whole repository [#2472](https://github.com/apache/datafusion-ballista/pull/2472) (NoahKusaba)
- docs(chaos): don't link to the feature-gated k8s module [#2506](https://github.com/apache/datafusion-ballista/pull/2506) (milenkovicm)
- docs: correct stale defaults, flags, and feature docs [#2478](https://github.com/apache/datafusion-ballista/pull/2478) (NoahKusaba)
- chore(release): release the Python client separately from the Rust crates [#2512](https://github.com/apache/datafusion-ballista/pull/2512) (andygrove)

**Other:**

- ci: install Rust via rustup in wheel build to fix release-build startup_failure [#2008](https://github.com/apache/datafusion-ballista/pull/2008) (andygrove)
- chore: Update project description to remove Apache Arrow [#2044](https://github.com/apache/datafusion-ballista/pull/2044) (andygrove)
- chore(deps): bump cryptography from 46.0.3 to 48.0.1 in /python [#2056](https://github.com/apache/datafusion-ballista/pull/2056) (dependabot[bot])
- chore(deps): bump pyjwt from 2.12.0 to 2.13.0 in /python [#2055](https://github.com/apache/datafusion-ballista/pull/2055) (dependabot[bot])
- chore(ci): Add a CI workflow that checks that the used CI actions and their versions are allowed [#2058](https://github.com/apache/datafusion-ballista/pull/2058) (martin-g)
- refactor(tui): Drop the header banner/logo [#2057](https://github.com/apache/datafusion-ballista/pull/2057) (martin-g)
- Add TPC-DS correctness testing [#2048](https://github.com/apache/datafusion-ballista/pull/2048) (andygrove)
- ci: skip expensive workflows for docs-only changes [#2073](https://github.com/apache/datafusion-ballista/pull/2073) (andygrove)
- chore(scheduler,executor): downgrade #2038 per-task traces to debug [#2104](https://github.com/apache/datafusion-ballista/pull/2104) (avantgardnerio)
- ci(tpch): add AQE-on smoke pass with multi-partition tasks [#2120](https://github.com/apache/datafusion-ballista/pull/2120) (avantgardnerio)
- Retryable IO errors misclassified when wrapped in DataFusionError::Shared [#2119](https://github.com/apache/datafusion-ballista/pull/2119) (goingforstudying-ctrl)
- chore: upgrade DataFusion to 54.1.0 [#2126](https://github.com/apache/datafusion-ballista/pull/2126) (andygrove)
- ci: allow the docs site deploy to be triggered manually [#2177](https://github.com/apache/datafusion-ballista/pull/2177) (andygrove)
- kll: add native Rust KLL quantile sketch [#2180](https://github.com/apache/datafusion-ballista/pull/2180) (avantgardnerio)
- chore(deps): bump arrow from 58.3.0 to 58.4.0 [#2178](https://github.com/apache/datafusion-ballista/pull/2178) (andygrove)
- chore(deps): bump quinn-proto from 0.11.14 to 0.11.16 [#2176](https://github.com/apache/datafusion-ballista/pull/2176) (dependabot[bot])
- ci: run TPC-DS SF1 with 1 and 4 partitions per task [#2197](https://github.com/apache/datafusion-ballista/pull/2197) (andygrove)
- chore: use taplo for Cargo.toml formatting [#2179](https://github.com/apache/datafusion-ballista/pull/2179) (villebro)
- test: add HA chaos harness that drives the scheduler's fault-tolerance paths on a real cluster [#2026](https://github.com/apache/datafusion-ballista/pull/2026) (andygrove)
- chore(deps): bump cryptography from 48.0.1 to 50.0.0 in /python [#2221](https://github.com/apache/datafusion-ballista/pull/2221) (dependabot[bot])
- ci: add h2o window suite CI job [#2230](https://github.com/apache/datafusion-ballista/pull/2230) (avantgardnerio)
- chore(docker): ca-certificates + h2o binary in benchmarks image [#2251](https://github.com/apache/datafusion-ballista/pull/2251) (avantgardnerio)
- refactor(scheduler): extract REST API wire types into ballista-api-types [#2256](https://github.com/apache/datafusion-ballista/pull/2256) (andygrove)
- refactor(core): use DataFusionError::External for structured task errors [#2276](https://github.com/apache/datafusion-ballista/pull/2276) (villebro)
- chore(deps): Track DataFusion git, pinned to the 55.0.0-rc2 tag [#2241](https://github.com/apache/datafusion-ballista/pull/2241) (andygrove)
- chore(deps): bump the DataFusion pin to the 55.0.0-rc3 tag [#2301](https://github.com/apache/datafusion-ballista/pull/2301) (andygrove)
- test: centralize physical plan snapshot assertions [#2297](https://github.com/apache/datafusion-ballista/pull/2297) (floze-the-genius)
- test(chaos): add a Kubernetes (kind) backend for the chaos harness [#2244](https://github.com/apache/datafusion-ballista/pull/2244) (akshaychitneni)
- chore: prune stale NOTICE.txt entries, add mini-redis attribution [#2300](https://github.com/apache/datafusion-ballista/pull/2300) (andygrove)
- chore(core): remove unused diagram module [#2355](https://github.com/apache/datafusion-ballista/pull/2355) (NoahKusaba)
- Revert "remove redundant logic from ballista logical codec (#2348)" [#2379](https://github.com/apache/datafusion-ballista/pull/2379) (andygrove)
- chore(deps): use the released DataFusion 55.0.0 crates from crates.io [#2375](https://github.com/apache/datafusion-ballista/pull/2375) (andygrove)
- ci: assign issues via take/untake comments [#2391](https://github.com/apache/datafusion-ballista/pull/2391) (andygrove)
- chore: require an approving review before merging [#2394](https://github.com/apache/datafusion-ballista/pull/2394) (andygrove)
- refactor(scheduler): share the constructor tail between SchedulerServer builders [#2398](https://github.com/apache/datafusion-ballista/pull/2398) (NoahKusaba)
- chore(deps): drop unused md-5, move insta to dev-deps [#2407](https://github.com/apache/datafusion-ballista/pull/2407) (NoahKusaba)
- chore: consolidate shared manifest metadata into the workspace [#2409](https://github.com/apache/datafusion-ballista/pull/2409) (NoahKusaba)
- test: dedupe shuffle reader test setup [#2415](https://github.com/apache/datafusion-ballista/pull/2415) (NoahKusaba)
- chore(ci): run cargo machete to catch unused dependencies [#2411](https://github.com/apache/datafusion-ballista/pull/2411) (7487)
- ci: share one cluster harness across the tpch, tpcds and h2o jobs [#2408](https://github.com/apache/datafusion-ballista/pull/2408) (NoahKusaba)
- bug: Prevent exec hang when task decode fails [#2016](https://github.com/apache/datafusion-ballista/pull/2016) (coderfender)
- chore(deps): bump DataFusion to 55.1.0 [#2461](https://github.com/apache/datafusion-ballista/pull/2461) (andygrove)
- ci: move the stale PR workflow where GitHub runs it [#2475](https://github.com/apache/datafusion-ballista/pull/2475) (NoahKusaba)
- ci: fix lint scripts that skipped checks or passed without running [#2473](https://github.com/apache/datafusion-ballista/pull/2473) (NoahKusaba)
- chore: remove leftovers inherited from Arrow [#2474](https://github.com/apache/datafusion-ballista/pull/2474) (NoahKusaba)
- refactor: move the Flight proxy service into ballista-core [#2495](https://github.com/apache/datafusion-ballista/pull/2495) (andygrove)
- test(chaos): run #2029 executor-loss scenarios on the k8s backend [#2425](https://github.com/apache/datafusion-ballista/pull/2425) (akshaychitneni)
- chore(deps): bump pyjwt from 2.13.0 to 2.15.0 in /python [#2515](https://github.com/apache/datafusion-ballista/pull/2515) (dependabot[bot])
- chore(ci): fix clippy errors from Rust 1.99 [#2519](https://github.com/apache/datafusion-ballista/pull/2519) (NoahKusaba)
- chore: bump Ballista version to 55.0.0 [#2524](https://github.com/apache/datafusion-ballista/pull/2524) (andygrove)
- chore(deps): bump urllib3 from 2.7.0 to 2.8.0 in /python [#2516](https://github.com/apache/datafusion-ballista/pull/2516) (dependabot[bot])

## Credits

Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.

```
   135	dependabot[bot]
    79	Andy Grove
    30	Brent Gardner
    24	NoahKusaba
     9	Daniël Heres
     7	Marko Milenković
     5	Akshay Chitneni
     5	Ville Brofeldt
     3	Jack Eadie
     3	Martin Grigorov
     3	Phillip LeBlanc
     3	alexander domenti
     2	Amit Vijapur
     2	Bhargava Vadlamani
     2	QuakeWang
     1	Anurag Tryambak Raut
     1	BingEdward
     1	Floze
     1	Joao Victor e Silva
     1	Li Zinuo
     1	Oleks V
     1	Stefan Wang
     1	Syed Talha Ahmed Gardazi
     1	debaditya
     1	goingforstudying-ctrl
```

Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.