Ballista Scheduler

Scheduler Identity

Each scheduler has a unique identifier used in cluster state, /api/state, and history event logs. Set it explicitly with --id when you want a stable value; otherwise the scheduler generates the same UUID-backed instance identity used for executor ids at startup. Use unique scheduler ids when multiple schedulers write to shared history or cluster state.

Executor ids follow the same model: each active executor registered with the same scheduler needs a unique id. Set it explicitly with --id when you want a stable value; otherwise the executor generates one at startup. A second active executor using an id that is already registered is rejected; the id can be reused after the previous executor is removed from scheduler state.

Executors report task status to the scheduler callback endpoint, which is built from --external-host and --bind-port.

Fetching Query Results

By default a client fetches the result partitions of a query directly from the executors that produced them, over Arrow Flight. This keeps the scheduler out of the data path, but it requires every client to have network access to every executor — which is not the case in isolated environments where only the scheduler is reachable.

For those deployments the scheduler can advertise a different address for clients to fetch results from, and can optionally host an Arrow Flight proxy itself.

Note: this is plain Arrow Flight, used to move result partitions. It is not Flight SQL, which was removed in Ballista 46.0.0 — the proxy only serves Ballista’s FetchPartition action, so generic Flight SQL or JDBC clients cannot use this endpoint.

Two independent options control this:

Option	Description
--enable-embedded-flight-proxy	Runs an Arrow Flight proxy inside the scheduler process, on the scheduler’s own host and port. The proxy forwards each fetch to the executor that owns the partition.
--advertise-flight-endpoint	The HOST:PORT address clients are told to fetch results from, instead of the executors. Use it to point clients at a load balancer or a standalone proxy.

The first controls whether a proxy runs; the second controls what clients are told. Setting both is a supported combination: the embedded proxy runs, and clients are pointed at the advertised address, which is how you put a load balancer in front of one or more schedulers.

To let clients fetch results through the scheduler itself:

ballista-scheduler --enable-embedded-flight-proxy

To point clients at a load balancer that fronts the schedulers:

ballista-scheduler --enable-embedded-flight-proxy \
                   --advertise-flight-endpoint ballista-flight.example.com:50050

Both options are disabled by default, and the embedded proxy should be enabled deliberately: it puts result traffic on the scheduler’s process and thread pool, competing with query planning and task scheduling. Under load this is a known source of scheduler congestion, so prefer a separate proxy — advertised with --advertise-flight-endpoint — for clusters where result volume is significant.

--advertise-flight-sql-endpoint is accepted as a deprecated alias of --advertise-flight-endpoint. Passing either flag with no value used to start the embedded proxy; that is deprecated too and logs a warning — use --enable-embedded-flight-proxy instead.

REST API

The scheduler also provides a REST API that allows jobs to be monitored.

These endpoints require the scheduler’s rest-api feature, which is enabled by default. Start the scheduler with --disable-rest-api to turn them off at runtime.

API	Method	Description
/api/openapi.json	GET	Return OpenAPI v3 specification document for the REST API.
/api/state	GET	Get the current scheduler state.
/api/version	GET	Get the scheduler’s Ballista version.
/api/executors	GET	Get a list of executors registered with the scheduler.
/api/executor/{executor_id}	GET	Get details for a single executor.
/api/jobs	GET	Get a list of jobs that have been submitted to the cluster.
/api/job/{job_id}	GET	Get a summary of a submitted job.
/api/job/{job_id}	PATCH	Cancel a currently running job
/api/job/{job_id}/config	GET	Get session configuration for a job.
/api/job/{job_id}/stages	GET	Get per-stage and per-task detail for a job.
/api/job/{job_id}/dot	GET	Produce a query plan in DOT (graphviz) format.
/api/job/{job_id}/dot_svg	GET	Produce a query plan in SVG format. (graphviz-support required)
/api/job/{job_id}/stage/{stage_id}/dot	GET	Produces stage plan in DOT (graphviz) format
/api/metrics	GET	Return current scheduler metric set

Two probe endpoints are always mounted, whether or not the rest-api feature is enabled:

API	Method	Description
/healthz	GET	Liveness. Always 200 OK while the process runs.
/readyz	GET	Readiness. 200 OK once at least --min-ready-executors executors are registered.

Web TUI Configuration

When the Scheduler is built with the rest-api feature, several command-line options control its integration with the Web TUI:

Option	Description
--web-tui-route	HTTP path that redirects to the hosted Web TUI. The default route is /.
--cors-allowed-origins	Comma-separated list of allowed CORS origins. By default, http://localhost:8080 and https://nightlies.apache.org are allowed.
--cors-allowed-methods	Comma-separated list of allowed CORS methods. By default, GET, PATCH, and OPTIONS are allowed.

For example, to expose the Web TUI redirect at http://localhost:50050/tui:

ballista-scheduler --web-tui-route /tui

The CORS options can be customized when hosting the Web TUI from another origin.