Upgrade Guides#

Ballista 56.0.0#

Note: Ballista 56.0.0 has not been released yet. The information provided in this section pertains to features and changes that have already been merged to the main branch and are awaiting release in this version.

Deprecations#

The cluster_config argument of BallistaSessionContext is deprecated#

cluster_config is still accepted, raises a DeprecationWarning, and will be removed in 57.0.0. Pass the same options with config instead:

# before
ctx = BallistaSessionContext(
    "df://localhost:50050",
    cluster_config={"ballista.job.name": "daily report"},
)

# after
ctx = BallistaSessionContext(
    "df://localhost:50050",
    config={"ballista.job.name": "daily report"},
)

Its options are merged into config, and take precedence if a key is set in both. As a result:

  • Options restricted by Ballista, for example datafusion.optimizer.prefer_hash_join, are always set to the Ballista value. Previously cluster_config overrode them on the local session.

  • cluster_config can not be combined with the new BallistaSessionConfig, doing so raises ValueError.

The analyze argument of explain_visual is deprecated#

explain_visual(df, analyze=True) in ballista.jupyter still works, but raises a DeprecationWarning. The argument only labelled the visualization: it never ran the query or added runtime statistics. For runtime statistics, use DataFusion’s explain instead:

# before
explain_visual(df, analyze=True)

# after
explain_visual(df)            # the plan, without running the query
df.explain(analyze=True)      # runs the query and prints runtime statistics

The scheduler’s embedded Flight proxy is deprecated#

--enable-embedded-flight-proxy (and SchedulerConfig::enable_embedded_flight_proxy) is deprecated and will be removed in 57.0.0. The proxy puts result traffic on the scheduler’s process, where it competes with query planning and task scheduling, and it scales only with the scheduler.

Deployments where clients cannot reach the executors should instead run the new ballista-result-service binary, a stateless Arrow Flight service that forwards each result fetch to the executor holding the partition, and point the scheduler’s --advertise-flight-endpoint at it, or at a load balancer in front of several replicas:

ballista-result-service --bind-port 50055
ballista-scheduler --advertise-flight-endpoint ballista-results.example.com:50055

For Kubernetes, see the example manifests in Serving Results to Clients Outside the Cluster.

Clients need no changes. The embedded proxy keeps working throughout 56.x, so you can compare the two and roll back without downgrading Ballista; the scheduler logs a warning at startup while it is enabled.

If you embed the scheduler and configure the proxy’s connections to executors with SchedulerConfig::use_tls and override_create_grpc_client_endpoint, for example for mTLS, run the result service from your own binary with ballista_result_service::serve. Its ResultServiceConfig takes the same two settings. The ballista-result-service binary connects to executors in plaintext, as the stock scheduler binary does.

Also deprecated:

  • SchedulerConfig::with_enable_embedded_flight_proxy

  • passing --advertise-flight-endpoint with no value, which starts the embedded proxy

Behavior changes#

Containers stop on SIGTERM#

The images’ entrypoints now exec their binaries, and each binary handles SIGTERM, so docker stop and pod termination reach the Ballista process. Before, the signal never did, and containers waited out their grace period to be killed. On Windows, Ctrl-Break and closing the console also start the shutdown, but Windows ends a console process about 5 seconds after its console closes, so a longer drain is cut short.

  • Executors now drain their running tasks before exiting, for up to --graceful-shutdown-timeout-seconds (10 by default). See Stopping executors for tasks that run longer.

  • The scheduler no longer exits at once on SIGTERM or Ctrl-C. It refuses new jobs with Unavailable, over gRPC and Flight SQL alike, lets its active jobs finish, and then drains its connections, all within --graceful-shutdown-timeout-seconds (25 by default). A second signal ends the shutdown at once, with an error. See Stopping the scheduler for how to size it and for how results fetched through the scheduler are served.

  • SchedulerConfig gains a graceful_shutdown_timeout_seconds field, so a struct literal that doesn’t end in ..Default::default() needs to set it. The public QueryStageSchedulerEvent::JobQueued gains an admission field.

  • scheduler_process::start_server_with_shutdown serves until the future that its shutdown argument returns completes, then shuts down as above; the binaries pass ballista_core::utils::shutdown_signal. start_server is unchanged and doesn’t handle signals.

  • The scheduler, the executor and the result service share one shutdown contract. After --graceful-shutdown-timeout-seconds they stop waiting and exit with an error, and 0 waits as long as the drain takes. The result service used to exit at once for 0, and to exit successfully on a timeout.

  • The CLI exits on SIGTERM.

  • ballista_executor::terminate::sig_term is deprecated. Use ballista_core::utils::terminate_signal, or shutdown_signal for Ctrl-C as well.

  • The standalone image stops its scheduler first, so that its active jobs can finish while the executor keeps running their tasks, and then its executor. The scheduler’s drain can outlast docker stop’s default of 10 seconds, so pass docker stop -t a time above --graceful-shutdown-timeout-seconds, 25 by default.

A range RepartitionExec in a submitted physical plan is planned as a shuffle#

The static planner used to drop every RepartitionExec that was not a hash repartition, so a range repartition followed by a per-partition sort (a distributed global sort) returned partitions that were sorted but not range-disjoint, with no error. A range repartition is now a stage boundary, and partition i of the stage above holds exactly range i.

One that must preserve its input order (RepartitionExec::with_preserve_order) is refused when the job is planned, because a shuffle cannot merge sorted inputs. Sort after the repartition instead.

Undecodable fetch tickets are rejected as InvalidArgument#

Executors, the embedded proxy, and the result service now share one implementation of partition fetches. A DoGet ticket that does not decode as a Ballista action is reported with the gRPC status InvalidArgument everywhere; executors used to report it as Internal.

Pull-staged executors no longer run more partitions than they have vcores#

Under pull-staged scheduling (--scheduler-policy pull-staged on the scheduler, --task-scheduling-policy pull-staged on executors), which standalone mode also uses, an executor counted each task as one vcore however many partitions the task held. With ballista.scheduler.max_partitions_per_task at its default of 0, an idle 8-vcore executor could take an 8-partition task, still report 7 vcores free, and be handed another task on its next poll. It could end up running several times more partitions than it has vcores, with per-task memory pools that added up to more than its memory budget, while executors that polled later could get no work.

An executor now holds a task’s vcores until the task finishes, as with push-staged scheduling. That includes the vcores of partitions that finish early, so a straggler partition leaves them idle. Set ballista.scheduler.max_partitions_per_task to a positive value to dispatch smaller tasks, which release their vcores sooner.

Stages split between tasks fail on a leaf the scheduler cannot restrict#

When a stage is split between tasks, each task gets a plan restricted to its own partitions. A leaf the scheduler did not know how to restrict used to be left as it was, so a task that ran partitions 4 and 5 of it ran its partitions 0 and 1, and the stage returned some partitions twice and others not at all, with no error.

The scheduler now fails the job instead. Such a leaf is accepted only in a task that runs all of its partitions, in order, so a retry of part of a stage can fail a job too. Plans made of Ballista’s own leaves (shuffle readers, file and in-memory scans, PlaceholderRowExec, EmptyExec) are unaffected, as are leaves read whole below a CoalescePartitionsExec or on a join’s build side. A DataSourceExec over any other source used to log a warning and run, and is now refused when its stage is split unless a resolver restricts it.

Whether a stage is split depends on how many task slots are free when it is scheduled, and on whether part of it is retried, so a query that holds such a leaf can pass on an idle or small cluster and fail on a busy one. To test a plan that holds nodes of your own, set ballista.scheduler.max_partitions_per_task to a small value, which splits every stage with more partitions than that.

Only leaves are checked. An interior node the scheduler does not know is left as it was while its children are restricted, because the scheduler cannot tell one that holds data per partition from one that does not, such as a FilterExec. An interior node of your own that holds data per partition still returns wrong results in a split stage unless a resolver restricts it.

The error names the node and what to implement. Under push-staged scheduling the job’s status says only that a task could not be prepared; the reason is in the scheduler’s log. To make nodes of your own restrictable, see Custom Physical Plan Nodes.

Nested-loop joins that emit build rows run as a single task#

Nested-loop joins of type Left, Full, LeftSemi, LeftAnti or LeftMark (the type EXPLAIN shows, after the planner picks the build side) used to return wrong results when their probe side was split between tasks, because each task emitted its own copy of the build rows. These joins now run as a single task, so queries that use them may be slower. Other nested-loop joins are unchanged.

API changes#

SortShuffleWriterExec accepts a range partitioning and a partition-id routing#

The generated SortShuffleWriterExecNode now says how the writer assigns rows to output partitions in a routing oneof: output_partitioning for a hash, partitioning for a range, and partition_id_routing for ids a client computed (see Partition-id shuffles). A oneof member is encoded like a plain field, so hash plans are encoded as before and plans from an older scheduler still decode.

In Rust, output_partitioning and partitioning are no longer fields of SortShuffleWriterExecNode. A struct literal of the message sets routing instead, for example routing: Some(sort_shuffle_writer_exec_node::Routing::OutputPartitioning(hash)), and code that reads the message matches on routing.

sort_shuffle::BufferedBatches is no longer public#

ballista_core::execution_plans::sort_shuffle::BufferedBatches, the sort-shuffle writer’s internal row buffer, is no longer exported. It had no use outside the writer. Code that used it directly should use SortShuffleWriterExec instead.

TaskManager::executor_lost returns an ExecutorLostOutcome#

ballista_scheduler::state::task_manager::TaskManager::executor_lost now returns an ExecutorLostOutcome instead of Vec<RunningTaskInfo>:

  • tasks_to_cancel: the running tasks to cancel, which is what it returned before.

  • failed_jobs: jobs that cannot survive the loss and must fail, with the reason and when each was queued: a pipelined job, or one that lost DataFrame::cache() data and whose graph cannot resume after it is rebuilt. The scheduler posts JobRunningFailed for each.

  • lost_inputs: jobs whose stages now wait for DataFrame::cache() data lost with the executor to be rebuilt, with the external stage ids of the lost reads. The scheduler posts ExternalInputLost for each.

QueryStageSchedulerEvent is #[non_exhaustive]#

ballista_scheduler::scheduler_server::event::QueryStageSchedulerEvent gains ExternalInputLost and ExternalInputReady, for recovering DataFrame::cache() data a running job loses, and is now #[non_exhaustive], so later events can be added without breaking code outside the crate. A match on it outside the crate needs a wildcard arm.

JobState gains new_context#

ballista_scheduler::cluster::JobState has a new method, new_context, which makes a context for work the scheduler starts on behalf of a running job, such as rebuilding DataFrame::cache() data it lost, from the job’s configuration. It has a default that calls create_or_update_session, so existing implementations keep compiling; one that records sessions should override it so a rebuild does not create or update a session. Such a context comes from the session builder, so anything registered directly on a hand-built context passed to SchedulerServer::submit_job is not in it; register it through override_session_builder instead.

PartitionSliceable covers any node, and embedders can register their own#

ballista_core::execution_plans::plan_algebra::PartitionSliceable is no longer limited to nodes with one child. slice_to_partitions(child, partitions) is now restrict_to_partitions(children, partitions), which receives all of the node’s children and also applies to leaves. Implementations of your own need the new signature; plan_algebra::single_child unwraps the one child of a single-input node.

Nodes of your own are made known to the scheduler with SchedulerConfig::with_partition_sliceable_resolver, which takes an implementation of the new plan_algebra::PartitionSliceableResolver trait and is consulted after Ballista’s own list. The resolver restricts the node itself, given its restricted children and the task’s partitions, so it also covers nodes that cannot implement PartitionSliceable, such as a DataSourceExec.

  • SchedulerConfig gains a partition_sliceable_resolver field, so a struct literal that doesn’t end in ..Default::default() needs to set it.

  • task_builder::restrict_plan_to_partitions takes a third argument, Option<&dyn PartitionSliceableResolver>. Pass None for none.

Partition locations hold an ExecutorEndpoint#

A partition location only has to say where to fetch from, so PartitionLocation::executor_meta and PipelinedPartitionLocation::executor_meta are now Arc<ExecutorEndpoint>, a new type with the executor’s id, host and port, shared between locations rather than copied into each one. ExecutorMetadata is unchanged, and is still what the scheduler records about a registered executor. To build an endpoint from one, use Arc::new(ExecutorEndpoint::from(&metadata)). Code that read grpc_port, specification or os_info from a location should look up the ExecutorMetadata by id instead.

Shuffle reader plan nodes now list each executor once and their locations refer to it by position. This changes the wire format between schedulers and executors, so BALLISTA_PROTOCOL_VERSION is now 4 and they must be upgraded together.

In the protobuf definitions, the ExecutorMetadata message is renamed ExecutorEndpoint and loses grpc_port, specification and os_info. Job statuses still set executor_meta on every location, so clients that read its id, host or port do not need to change.

The disable-stage-plan-cache feature of ballista-scheduler is removed#

The scheduler stopped caching encoded stage plans in 55.0.0 and now encodes each task’s plan separately, so the feature had no effect. Remove it from the features list of your ballista-scheduler dependency. Behavior is unchanged.

The spark-compat feature is removed#

The spark-compat feature of ballista-core, ballista-scheduler and ballista-executor is removed, along with their datafusion-spark dependency, the remote-spark-functions example and the Spark-Compatible Functions page of the user guide. Remove spark-compat from the features list of your Ballista dependencies.

The feature registered the functions of datafusion-spark in every session. To keep using them, depend on datafusion-spark yourself and register its all_default_scalar_functions(), all_default_aggregate_functions() and all_default_window_functions() everywhere a plan that calls them is decoded:

  • on the client, on the SessionState the SessionContext is created from;

  • on the scheduler, in the session built by SchedulerConfig::override_session_builder;

  • on the executor, in the BallistaFunctionRegistry passed as ExecutorProcessConfig::override_function_registry.

The scheduler’s REST endpoint for its state, and the scheduler info popup of the TUI, no longer report a spark_support field.

Python client changes#

Summary#

  • The Ballista Python client now plugs into DataFusion Python natively. It registers a custom query planner on a standard datafusion.SessionContext, instead of wrapping DataFusion classes.

  • DistributedDataFrame and the metaclasses that wrapped SessionContext and DataFrame are removed. Queries return a plain datafusion.DataFrame.

  • BallistaSessionContext is now a factory function that returns a datafusion.SessionContext, not a class.

  • explain_visual and collect_with_progress are no longer DataFrame methods. They are functions in ballista.jupyter.

  • Ballista settings are registered as a DataFusion config extension, exposed as BallistaConfigExtension.

  • A Ballista client implemented in Python, BallistaClient, is added. Pass it to BallistaSessionContext(client=...), or set ballista.python.client_impl to python in config, to use it instead of the native Rust client. Its dependencies are not installed by default, they come with the python-client extra.

The Python cluster client is an optional extra#

BallistaClient is a Ballista client implemented in Python. It talks to the scheduler over gRPC and fetches results over Arrow Flight. It needs grpcio, protobuf>=7.35.1,<8 and pyarrow.flight. To keep these out of environments that don’t use the client, they come with the python-client extra and are not installed by default:

pip install 'ballista[python-client]'

BallistaSessionContext uses it when it is passed as client:

from ballista import BallistaClient, BallistaSessionContext

ctx = BallistaSessionContext(client=BallistaClient("df://localhost:50050"))

or, without creating the client, when ballista.python.client_impl is set to python in config. The client is then created for the context’s address with the ballista.* options of config:

ctx = BallistaSessionContext(
    "df://localhost:50050",
    config={"ballista.python.client_impl": "python", "ballista.client.use_tls": True},
)

ballista.python.client_impl is used only by Python to pick the client, it is not sent to the cluster. It can only be set in config, SET ballista.python.client_impl fails with an unknown key error.

A passed client connects to the scheduler with the ballista.client.use_tls, ballista.client.grpc_max_message_size and ballista.client.grpc_http2_keepalive_interval_seconds options of its own config. If the context’s config sets them to other values, a UserWarning is raised; set them on the client instead. Other ballista.* options of the context’s config apply to the client’s queries.

Without client, BallistaSessionContext uses the native Rust client and does not need the extra. import ballista works without it too, because the client is only imported when one of its classes is first used. Without the extra, that use raises an ImportError that names the extra.

DataFusion Python is supported natively#

Earlier releases subclassed SessionContext and DataFrame. Every method that executed a plan was intercepted, the logical plan was serialized, and the plan was resubmitted through a separate, Ballista-backed session. Methods the wrapper did not know about ran locally, and every execution created a new Ballista session.

In 56.0.0, BallistaSessionContext installs a Ballista query planner and extension codecs on a standard DataFusion Python SessionContext, using DataFusion’s FFI. DataFusion plans each query as usual, and the planner sends the physical plan to the scheduler. Any operation that executes a plan runs on the cluster, including collect, show, to_pandas, write_parquet and EXPLAIN ANALYZE. No wrapper is involved.

DistributedDataFrame is removed#

ballista.DistributedDataFrame no longer exists, and ballista.extension no longer exports it. ctx.sql(...), ctx.read_parquet(...) and the other context methods return a regular datafusion.DataFrame.

Action required: remove imports of DistributedDataFrame, and replace type hints and isinstance checks with datafusion.DataFrame:

# before
from ballista import DistributedDataFrame

def run(df: DistributedDataFrame): ...

# after
from datafusion import DataFrame

def run(df: DataFrame): ...

BallistaSessionContext is a factory function#

BallistaSessionContext keeps its name and call signature, so existing code that creates a context does not change:

from ballista import BallistaSessionContext

ctx = BallistaSessionContext("df://localhost:50050")

It is now a function that returns a datafusion.SessionContext. It is not a class. Code that depends on it being a class breaks:

  • isinstance(ctx, BallistaSessionContext) raises TypeError. Use isinstance(ctx, datafusion.SessionContext).

  • Subclassing BallistaSessionContext is no longer possible. Create a context and pass it to your own code, or wrap it.

  • ctx.session_id is no longer a property. Call ctx.session_id(), as on any DataFusion SessionContext.

  • The ctx.address and ctx.cluster_config attributes are removed.

  • ctx.get_tables() is removed. Use the DataFusion catalog API, for example ctx.catalog().schema("public").table_names().

Configuration changes#

All settings are now passed with config, which is used by the local session and sent to the scheduler-side session with every query. Previously a SessionConfig passed as config was used locally only. The cluster_config argument is deprecated and will be removed in 57.0.0, see Deprecations.

config accepts:

  • a dictionary of datafusion.* and ballista.* options, with str, int, float or bool values. Booleans are converted to "true" and "false". Any other key prefix, or an unknown ballista.* key, raises ValueError.

  • a datafusion.SessionConfig.

  • a ballista.BallistaSessionConfig, a new builder with typed with_* setters for Ballista options and set for any datafusion.* or ballista.* option.

# before
ctx = BallistaSessionContext(
    "df://localhost:50050",
    cluster_config={
        "datafusion.execution.target_partitions": 16,
        "ballista.job.name": "daily report",
        "ballista.planner.adaptive.enabled": True,
    },
)

# after
ctx = BallistaSessionContext(
    "df://localhost:50050",
    config={
        "datafusion.execution.target_partitions": 16,
        "ballista.job.name": "daily report",
        "ballista.planner.adaptive.enabled": True,
    },
)

# or
from ballista import BallistaSessionConfig

ctx = BallistaSessionContext(
    "df://localhost:50050",
    config=BallistaSessionConfig()
    .with_job_name("daily report")
    .with_adaptive_query_planner(True)
    .with_target_partitions(16),
)

BallistaSessionConfig.build() returns a regular SessionConfig, so it can also be used to create a plain DataFusion SessionContext.

Ballista settings are registered on the session as a DataFusion config extension, available as ballista.BallistaConfigExtension. As a result, ballista.* keys can be changed with SQL SET, for example SET ballista.job.name = 'nightly'.

The local session now starts from Ballista’s DataFusion defaults, the same defaults the scheduler uses. Some DataFusion options are restricted by Ballista (for example datafusion.optimizer.prefer_hash_join and datafusion.optimizer.enable_round_robin_repartition). These options are always set to the Ballista value, even if you set them in config. Other Ballista defaults, such as datafusion.optimizer.hash_join_single_partition_threshold, can still be overridden when config is passed as a dictionary or set with BallistaSessionConfig.set. Options inside a passed SessionConfig can not be inspected, so all of them are overridden with Ballista defaults.

Jupyter support changes#

DataFrame.explain_visual() and DataFrame.collect_with_progress() were methods of DistributedDataFrame and are removed with it. They are now functions in ballista.jupyter that take the DataFrame as their first argument:

from ballista.jupyter import collect_with_progress, explain_visual

df = ctx.sql("SELECT * FROM orders WHERE amount > 100")

# before: df.explain_visual()
explain_visual(df)
explain_visual(df).save("plan.svg")

# before: df.collect_with_progress()
batches = collect_with_progress(df)
batches = collect_with_progress(df, callback=lambda status, progress: print(status))

The parameters and return values are unchanged, except that analyze=True is now deprecated, see Deprecations.

The %sql magic returns a datafusion.DataFrame instead of a DistributedDataFrame. %ballista tables uses the DataFusion catalog directly, since ctx.get_tables() is removed. %ballista status shows the session id from ctx.session_id(). The magic commands themselves are unchanged.