Upgrade Guides#
Ballista 56.0.0#
Note: Ballista 56.0.0 has not been released yet. The information provided
in this section pertains to features and changes that have already been merged
to the main branch and are awaiting release in this version.
Deprecations#
The cluster_config argument of BallistaSessionContext is deprecated#
cluster_config is still accepted, raises a DeprecationWarning, and will be
removed in 57.0.0. Pass the same options with config instead:
# before
ctx = BallistaSessionContext(
"df://localhost:50050",
cluster_config={"ballista.job.name": "daily report"},
)
# after
ctx = BallistaSessionContext(
"df://localhost:50050",
config={"ballista.job.name": "daily report"},
)
Its options are merged into config, and take precedence if a key is set in
both. As a result:
Options restricted by Ballista, for example
datafusion.optimizer.prefer_hash_join, are always set to the Ballista value. Previouslycluster_configoverrode them on the local session.cluster_configcan not be combined with the newBallistaSessionConfig, doing so raisesValueError.
The analyze argument of explain_visual is deprecated#
explain_visual(df, analyze=True) in ballista.jupyter still works, but
raises a DeprecationWarning. The argument only labelled the visualization: it
never ran the query or added runtime statistics. For runtime statistics, use
DataFusion’s explain instead:
# before
explain_visual(df, analyze=True)
# after
explain_visual(df) # the plan, without running the query
df.explain(analyze=True) # runs the query and prints runtime statistics
The scheduler’s embedded Flight proxy is deprecated#
--enable-embedded-flight-proxy (and SchedulerConfig::enable_embedded_flight_proxy)
is deprecated and will be removed in 57.0.0. The proxy puts result traffic on the
scheduler’s process, where it competes with query planning and task scheduling,
and it scales only with the scheduler.
Deployments where clients cannot reach the executors should instead run the new
ballista-result-service binary, a stateless Arrow Flight service that forwards
each result fetch to the executor holding the partition, and point the
scheduler’s --advertise-flight-endpoint at it, or at a load balancer in front
of several replicas:
ballista-result-service --bind-port 50055
ballista-scheduler --advertise-flight-endpoint ballista-results.example.com:50055
For Kubernetes, see the example manifests in Serving Results to Clients Outside the Cluster.
Clients need no changes. The embedded proxy keeps working throughout 56.x, so you can compare the two and roll back without downgrading Ballista; the scheduler logs a warning at startup while it is enabled.
If you embed the scheduler and configure the proxy’s connections to executors
with SchedulerConfig::use_tls and override_create_grpc_client_endpoint, for
example for mTLS, run the result service from your own binary with
ballista_result_service::serve. Its ResultServiceConfig takes the same two
settings. The ballista-result-service binary connects to executors in
plaintext, as the stock scheduler binary does.
Also deprecated:
SchedulerConfig::with_enable_embedded_flight_proxypassing
--advertise-flight-endpointwith no value, which starts the embedded proxy
Behavior changes#
Containers stop on SIGTERM#
The images’ entrypoints now exec their binaries, and each binary handles
SIGTERM, so docker stop and pod termination reach the Ballista process.
Before, the signal never did, and containers waited out their grace period to
be killed. On Windows, Ctrl-Break and closing the console also start the
shutdown, but Windows ends a console process about 5 seconds after its console
closes, so a longer drain is cut short.
Executors now drain their running tasks before exiting, for up to
--graceful-shutdown-timeout-seconds(10 by default). See Stopping executors for tasks that run longer.The scheduler no longer exits at once on SIGTERM or Ctrl-C. It refuses new jobs with
Unavailable, over gRPC and Flight SQL alike, lets its active jobs finish, and then drains its connections, all within--graceful-shutdown-timeout-seconds(25 by default). A second signal ends the shutdown at once, with an error. See Stopping the scheduler for how to size it and for how results fetched through the scheduler are served.SchedulerConfiggains agraceful_shutdown_timeout_secondsfield, so a struct literal that doesn’t end in..Default::default()needs to set it. The publicQueryStageSchedulerEvent::JobQueuedgains anadmissionfield.scheduler_process::start_server_with_shutdownserves until the future that itsshutdownargument returns completes, then shuts down as above; the binaries passballista_core::utils::shutdown_signal.start_serveris unchanged and doesn’t handle signals.The scheduler, the executor and the result service share one shutdown contract. After
--graceful-shutdown-timeout-secondsthey stop waiting and exit with an error, and0waits as long as the drain takes. The result service used to exit at once for0, and to exit successfully on a timeout.The CLI exits on SIGTERM.
ballista_executor::terminate::sig_termis deprecated. Useballista_core::utils::terminate_signal, orshutdown_signalfor Ctrl-C as well.The standalone image stops its scheduler first, so that its active jobs can finish while the executor keeps running their tasks, and then its executor. The scheduler’s drain can outlast
docker stop’s default of 10 seconds, so passdocker stop -ta time above--graceful-shutdown-timeout-seconds, 25 by default.
A range RepartitionExec in a submitted physical plan is planned as a shuffle#
The static planner used to drop every RepartitionExec that was not a hash
repartition, so a range repartition followed by a per-partition sort (a
distributed global sort) returned partitions that were sorted but not
range-disjoint, with no error. A range repartition is now a stage boundary, and
partition i of the stage above holds exactly range i.
One that must preserve its input order (RepartitionExec::with_preserve_order)
is refused when the job is planned, because a shuffle cannot merge sorted
inputs. Sort after the repartition instead.
Undecodable fetch tickets are rejected as InvalidArgument#
Executors, the embedded proxy, and the result service now share one
implementation of partition fetches. A DoGet ticket that does not decode as a
Ballista action is reported with the gRPC status InvalidArgument everywhere;
executors used to report it as Internal.
Pull-staged executors no longer run more partitions than they have vcores#
Under pull-staged scheduling (--scheduler-policy pull-staged on the scheduler,
--task-scheduling-policy pull-staged on executors), which standalone mode also
uses, an executor counted each task as one vcore however many partitions the
task held. With ballista.scheduler.max_partitions_per_task at its default of
0, an idle 8-vcore executor could take an 8-partition task, still report 7
vcores free, and be handed another task on its next poll. It could end up
running several times more partitions than it has vcores, with per-task memory
pools that added up to more than its memory budget, while executors that polled
later could get no work.
An executor now holds a task’s vcores until the task finishes, as with
push-staged scheduling. That includes the vcores of partitions that finish
early, so a straggler partition leaves them idle. Set
ballista.scheduler.max_partitions_per_task to a positive value to dispatch
smaller tasks, which release their vcores sooner.
Stages split between tasks fail on a leaf the scheduler cannot restrict#
When a stage is split between tasks, each task gets a plan restricted to its own partitions. A leaf the scheduler did not know how to restrict used to be left as it was, so a task that ran partitions 4 and 5 of it ran its partitions 0 and 1, and the stage returned some partitions twice and others not at all, with no error.
The scheduler now fails the job instead. Such a leaf is accepted only in a task
that runs all of its partitions, in order, so a retry of part of a stage can
fail a job too. Plans made of Ballista’s own leaves (shuffle readers, file and
in-memory scans, PlaceholderRowExec, EmptyExec) are unaffected, as are
leaves read whole below a CoalescePartitionsExec or on a join’s build side. A
DataSourceExec over any other source used to log a warning and run, and is
now refused when its stage is split unless a resolver restricts it.
Whether a stage is split depends on how many task slots are free when it is
scheduled, and on whether part of it is retried, so a query that holds such a
leaf can pass on an idle or small cluster and fail on a busy one. To test a plan
that holds nodes of your own, set ballista.scheduler.max_partitions_per_task
to a small value, which splits every stage with more partitions than that.
Only leaves are checked. An interior node the scheduler does not know is left as
it was while its children are restricted, because the scheduler cannot tell one
that holds data per partition from one that does not, such as a FilterExec. An
interior node of your own that holds data per partition still returns wrong
results in a split stage unless a resolver restricts it.
The error names the node and what to implement. Under push-staged scheduling the job’s status says only that a task could not be prepared; the reason is in the scheduler’s log. To make nodes of your own restrictable, see Custom Physical Plan Nodes.
Nested-loop joins that emit build rows run as a single task#
Nested-loop joins of type Left, Full, LeftSemi, LeftAnti or LeftMark
(the type EXPLAIN shows, after the planner picks the build side) used to
return wrong results when their probe side was split between tasks, because
each task emitted its own copy of the build rows. These joins now run as a
single task, so queries that use them may be slower. Other nested-loop joins
are unchanged.
API changes#
SortShuffleWriterExec accepts a range partitioning and a partition-id routing#
The generated SortShuffleWriterExecNode now says how the writer assigns rows
to output partitions in a routing oneof: output_partitioning for a hash,
partitioning for a range, and partition_id_routing for ids a client computed
(see Partition-id shuffles).
A oneof member is encoded like a plain field, so hash plans are encoded as
before and plans from an older scheduler still decode.
In Rust, output_partitioning and partitioning are no longer fields of
SortShuffleWriterExecNode. A struct literal of the message sets routing
instead, for example
routing: Some(sort_shuffle_writer_exec_node::Routing::OutputPartitioning(hash)),
and code that reads the message matches on routing.
sort_shuffle::BufferedBatches is no longer public#
ballista_core::execution_plans::sort_shuffle::BufferedBatches, the sort-shuffle
writer’s internal row buffer, is no longer exported. It had no use outside the
writer. Code that used it directly should use SortShuffleWriterExec instead.
TaskManager::executor_lost returns an ExecutorLostOutcome#
ballista_scheduler::state::task_manager::TaskManager::executor_lost now returns
an ExecutorLostOutcome instead of Vec<RunningTaskInfo>:
tasks_to_cancel: the running tasks to cancel, which is what it returned before.failed_jobs: jobs that cannot survive the loss and must fail, with the reason and when each was queued: a pipelined job, or one that lostDataFrame::cache()data and whose graph cannot resume after it is rebuilt. The scheduler postsJobRunningFailedfor each.lost_inputs: jobs whose stages now wait forDataFrame::cache()data lost with the executor to be rebuilt, with the external stage ids of the lost reads. The scheduler postsExternalInputLostfor each.
QueryStageSchedulerEvent is #[non_exhaustive]#
ballista_scheduler::scheduler_server::event::QueryStageSchedulerEvent gains
ExternalInputLost and ExternalInputReady, for recovering
DataFrame::cache() data a running job loses, and is now #[non_exhaustive],
so later events can be added without breaking code outside the crate. A match
on it outside the crate needs a wildcard arm.
JobState gains new_context#
ballista_scheduler::cluster::JobState has a new method, new_context, which
makes a context for work the scheduler starts on behalf of a running job, such
as rebuilding DataFrame::cache() data it lost, from the job’s configuration.
It has a default that calls create_or_update_session, so existing
implementations keep compiling; one that records sessions should override it so
a rebuild does not create or update a session. Such a context comes from the
session builder, so anything registered directly on a hand-built context passed
to SchedulerServer::submit_job is not in it; register it through
override_session_builder instead.
PartitionSliceable covers any node, and embedders can register their own#
ballista_core::execution_plans::plan_algebra::PartitionSliceable is no longer
limited to nodes with one child. slice_to_partitions(child, partitions) is now
restrict_to_partitions(children, partitions), which receives all of the node’s
children and also applies to leaves. Implementations of your own need the new
signature; plan_algebra::single_child unwraps the one child of a single-input
node.
Nodes of your own are made known to the scheduler with
SchedulerConfig::with_partition_sliceable_resolver, which takes an
implementation of the new plan_algebra::PartitionSliceableResolver trait and
is consulted after Ballista’s own list. The resolver restricts the node itself,
given its restricted children and the task’s partitions, so it also covers
nodes that cannot implement PartitionSliceable, such as a DataSourceExec.
SchedulerConfiggains apartition_sliceable_resolverfield, so a struct literal that doesn’t end in..Default::default()needs to set it.task_builder::restrict_plan_to_partitionstakes a third argument,Option<&dyn PartitionSliceableResolver>. PassNonefor none.
Partition locations hold an ExecutorEndpoint#
A partition location only has to say where to fetch from, so
PartitionLocation::executor_meta and
PipelinedPartitionLocation::executor_meta are now Arc<ExecutorEndpoint>, a
new type with the executor’s id, host and port, shared between locations
rather than copied into each one. ExecutorMetadata is unchanged, and is still
what the scheduler records about a registered executor. To build an endpoint
from one, use Arc::new(ExecutorEndpoint::from(&metadata)). Code that read
grpc_port, specification or os_info from a location should look up the
ExecutorMetadata by id instead.
Shuffle reader plan nodes now list each executor once and their locations refer
to it by position. This changes the wire format between schedulers and
executors, so BALLISTA_PROTOCOL_VERSION is now 4 and they must be upgraded
together.
In the protobuf definitions, the ExecutorMetadata message is renamed
ExecutorEndpoint and loses grpc_port, specification and os_info. Job
statuses still set executor_meta on every location, so clients that read its
id, host or port do not need to change.
The disable-stage-plan-cache feature of ballista-scheduler is removed#
The scheduler stopped caching encoded stage plans in 55.0.0 and now encodes
each task’s plan separately, so the feature had no effect. Remove it from the
features list of your ballista-scheduler dependency. Behavior is unchanged.
The spark-compat feature is removed#
The spark-compat feature of ballista-core, ballista-scheduler and
ballista-executor is removed, along with their datafusion-spark dependency,
the remote-spark-functions example and the Spark-Compatible Functions page of
the user guide. Remove spark-compat from the features list of your Ballista
dependencies.
The feature registered the functions of
datafusion-spark in every
session. To keep using them, depend on datafusion-spark yourself and register
its all_default_scalar_functions(), all_default_aggregate_functions() and
all_default_window_functions() everywhere a plan that calls them is decoded:
on the client, on the
SessionStatetheSessionContextis created from;on the scheduler, in the session built by
SchedulerConfig::override_session_builder;on the executor, in the
BallistaFunctionRegistrypassed asExecutorProcessConfig::override_function_registry.
The scheduler’s REST endpoint for its state, and the scheduler info popup of
the TUI, no longer report a spark_support field.
Python client changes#
Summary#
The Ballista Python client now plugs into DataFusion Python natively. It registers a custom query planner on a standard
datafusion.SessionContext, instead of wrapping DataFusion classes.DistributedDataFrameand the metaclasses that wrappedSessionContextandDataFrameare removed. Queries return a plaindatafusion.DataFrame.BallistaSessionContextis now a factory function that returns adatafusion.SessionContext, not a class.explain_visualandcollect_with_progressare no longerDataFramemethods. They are functions inballista.jupyter.Ballista settings are registered as a DataFusion config extension, exposed as
BallistaConfigExtension.A Ballista client implemented in Python,
BallistaClient, is added. Pass it toBallistaSessionContext(client=...), or setballista.python.client_impltopythoninconfig, to use it instead of the native Rust client. Its dependencies are not installed by default, they come with thepython-clientextra.
The Python cluster client is an optional extra#
BallistaClient is a Ballista client implemented in Python. It talks to the
scheduler over gRPC and fetches results over Arrow Flight. It needs grpcio,
protobuf>=7.35.1,<8 and pyarrow.flight. To keep these out of environments that don’t use the client,
they come with the python-client extra and are not installed by default:
pip install 'ballista[python-client]'
BallistaSessionContext uses it when it is passed as client:
from ballista import BallistaClient, BallistaSessionContext
ctx = BallistaSessionContext(client=BallistaClient("df://localhost:50050"))
or, without creating the client, when ballista.python.client_impl is set to python in config.
The client is then created for the context’s address with the ballista.* options of config:
ctx = BallistaSessionContext(
"df://localhost:50050",
config={"ballista.python.client_impl": "python", "ballista.client.use_tls": True},
)
ballista.python.client_impl is used only by Python to pick the client, it is not sent to the cluster.
It can only be set in config, SET ballista.python.client_impl fails with an unknown key error.
A passed client connects to the scheduler with the ballista.client.use_tls,
ballista.client.grpc_max_message_size and ballista.client.grpc_http2_keepalive_interval_seconds
options of its own config. If the context’s config sets them to other values, a UserWarning
is raised; set them on the client instead. Other ballista.* options of the context’s config
apply to the client’s queries.
Without client, BallistaSessionContext uses the native Rust client and
does not need the extra. import ballista works without it too, because the
client is only imported when one of its classes is first used. Without the extra, that use
raises an ImportError that names the extra.
DataFusion Python is supported natively#
Earlier releases subclassed SessionContext and DataFrame. Every method that
executed a plan was intercepted, the logical plan was serialized, and the plan
was resubmitted through a separate, Ballista-backed session. Methods the wrapper
did not know about ran locally, and every execution created a new Ballista
session.
In 56.0.0, BallistaSessionContext installs a Ballista query planner and
extension codecs on a standard DataFusion Python SessionContext, using
DataFusion’s FFI. DataFusion plans each query as usual, and the planner sends
the physical plan to the scheduler. Any operation that executes a plan runs on
the cluster, including collect, show, to_pandas, write_parquet and
EXPLAIN ANALYZE. No wrapper is involved.
DistributedDataFrame is removed#
ballista.DistributedDataFrame no longer exists, and ballista.extension no
longer exports it. ctx.sql(...), ctx.read_parquet(...) and the other
context methods return a regular datafusion.DataFrame.
Action required: remove imports of DistributedDataFrame, and replace type
hints and isinstance checks with datafusion.DataFrame:
# before
from ballista import DistributedDataFrame
def run(df: DistributedDataFrame): ...
# after
from datafusion import DataFrame
def run(df: DataFrame): ...
BallistaSessionContext is a factory function#
BallistaSessionContext keeps its name and call signature, so existing code
that creates a context does not change:
from ballista import BallistaSessionContext
ctx = BallistaSessionContext("df://localhost:50050")
It is now a function that returns a datafusion.SessionContext. It is not a
class. Code that depends on it being a class breaks:
isinstance(ctx, BallistaSessionContext)raisesTypeError. Useisinstance(ctx, datafusion.SessionContext).Subclassing
BallistaSessionContextis no longer possible. Create a context and pass it to your own code, or wrap it.ctx.session_idis no longer a property. Callctx.session_id(), as on any DataFusionSessionContext.The
ctx.addressandctx.cluster_configattributes are removed.ctx.get_tables()is removed. Use the DataFusion catalog API, for examplectx.catalog().schema("public").table_names().
Configuration changes#
All settings are now passed with config, which is used by the local session
and sent to the scheduler-side session with every query. Previously a
SessionConfig passed as config was used locally only. The cluster_config
argument is deprecated and will be removed in 57.0.0, see
Deprecations.
config accepts:
a dictionary of
datafusion.*andballista.*options, withstr,int,floatorboolvalues. Booleans are converted to"true"and"false". Any other key prefix, or an unknownballista.*key, raisesValueError.a
datafusion.SessionConfig.a
ballista.BallistaSessionConfig, a new builder with typedwith_*setters for Ballista options andsetfor anydatafusion.*orballista.*option.
# before
ctx = BallistaSessionContext(
"df://localhost:50050",
cluster_config={
"datafusion.execution.target_partitions": 16,
"ballista.job.name": "daily report",
"ballista.planner.adaptive.enabled": True,
},
)
# after
ctx = BallistaSessionContext(
"df://localhost:50050",
config={
"datafusion.execution.target_partitions": 16,
"ballista.job.name": "daily report",
"ballista.planner.adaptive.enabled": True,
},
)
# or
from ballista import BallistaSessionConfig
ctx = BallistaSessionContext(
"df://localhost:50050",
config=BallistaSessionConfig()
.with_job_name("daily report")
.with_adaptive_query_planner(True)
.with_target_partitions(16),
)
BallistaSessionConfig.build() returns a regular SessionConfig, so it can
also be used to create a plain DataFusion SessionContext.
Ballista settings are registered on the session as a DataFusion config
extension, available as ballista.BallistaConfigExtension. As a result,
ballista.* keys can be changed with SQL SET, for example
SET ballista.job.name = 'nightly'.
The local session now starts from Ballista’s DataFusion defaults, the same
defaults the scheduler uses. Some DataFusion options are restricted by Ballista
(for example datafusion.optimizer.prefer_hash_join and
datafusion.optimizer.enable_round_robin_repartition). These options are
always set to the Ballista value, even if you set them in config. Other
Ballista defaults, such as
datafusion.optimizer.hash_join_single_partition_threshold, can still be
overridden when config is passed as a dictionary or set with
BallistaSessionConfig.set. Options inside a passed SessionConfig can not be
inspected, so all of them are overridden with Ballista defaults.
Jupyter support changes#
DataFrame.explain_visual() and DataFrame.collect_with_progress() were
methods of DistributedDataFrame and are removed with it. They are now
functions in ballista.jupyter that take the DataFrame as their first argument:
from ballista.jupyter import collect_with_progress, explain_visual
df = ctx.sql("SELECT * FROM orders WHERE amount > 100")
# before: df.explain_visual()
explain_visual(df)
explain_visual(df).save("plan.svg")
# before: df.collect_with_progress()
batches = collect_with_progress(df)
batches = collect_with_progress(df, callback=lambda status, progress: print(status))
The parameters and return values are unchanged, except that analyze=True is
now deprecated, see Deprecations.
The %sql magic returns a datafusion.DataFrame instead of a
DistributedDataFrame. %ballista tables uses the DataFusion catalog
directly, since ctx.get_tables() is removed. %ballista status shows the
session id from ctx.session_id(). The magic commands themselves are
unchanged.