Upgrade Guides#

Ballista 56.0.0#

Note: Ballista 56.0.0 has not been released yet. The information provided in this section pertains to features and changes that have already been merged to the main branch and are awaiting release in this version.

Deprecations#

The scheduler’s embedded Flight proxy is deprecated#

--enable-embedded-flight-proxy (and SchedulerConfig::enable_embedded_flight_proxy) is deprecated and will be removed in 57.0.0. The proxy puts result traffic on the scheduler’s process, where it competes with query planning and task scheduling, and it scales only with the scheduler.

Deployments where clients cannot reach the executors should instead run the new ballista-result-service binary, a stateless Arrow Flight service that forwards each result fetch to the executor holding the partition, and point the scheduler’s --advertise-flight-endpoint at it, or at a load balancer in front of several replicas:

ballista-result-service --bind-port 50055
ballista-scheduler --advertise-flight-endpoint ballista-results.example.com:50055

For Kubernetes, see the example manifests in Serving Results to Clients Outside the Cluster.

Clients need no changes. The embedded proxy keeps working throughout 56.x, so you can compare the two and roll back without downgrading Ballista; the scheduler logs a warning at startup while it is enabled.

If you embed the scheduler and configure the proxy’s connections to executors with SchedulerConfig::use_tls and override_create_grpc_client_endpoint, for example for mTLS, run the result service from your own binary with ballista_result_service::serve. Its ResultServiceConfig takes the same two settings. The ballista-result-service binary connects to executors in plaintext, as the stock scheduler binary does.

Also deprecated:

  • SchedulerConfig::with_enable_embedded_flight_proxy

  • passing --advertise-flight-endpoint with no value, which starts the embedded proxy

Behavior changes#

Containers stop on SIGTERM#

The images’ entrypoints now exec their binaries, and each binary handles SIGTERM, so docker stop and pod termination reach the Ballista process. Before, the signal never did, and containers waited out their grace period to be killed.

  • Executors now drain their running tasks before exiting. If tasks can run longer than 30 seconds, raise the executor pods’ terminationGracePeriodSeconds and the scheduler’s --executor-termination-grace-period to match.

  • The scheduler exits on SIGTERM or Ctrl-C, without draining its connections.

  • The CLI exits on SIGTERM.

  • The standalone image stops its executor first, then its scheduler.

Undecodable fetch tickets are rejected as InvalidArgument#

Executors, the embedded proxy, and the result service now share one implementation of partition fetches. A DoGet ticket that does not decode as a Ballista action is reported with the gRPC status InvalidArgument everywhere; executors used to report it as Internal.