DataFrame Cache Design (DataFrame::cache())#
Goal#
df.cache() marks a DataFrame to be computed once and reused by later
actions. In Ballista a cached dataset is shuffle output kept (pinned) past job
cleanup, plus a scheduler-side record of where it lives. It reuses what
Ballista already has — shuffle files, the shuffle reader, and stage recovery —
rather than adding a cache store or a recovery path of its own.
The cache is a hint, not a correctness dependency: a cached node keeps its subplan (lineage), so lost cache data is recomputed, never a wrong or partial answer. A query that loses it mid-run rebuilds it and carries on, or, where it cannot, fails with a clear error and the retry rebuilds it (Recovery).
End-to-end flow#
df.cache() is lazy: it wraps the DataFrame’s logical plan in a
BallistaCacheNode with a fresh cache_id (a per-call UUID). Any plan later
submitted that still contains the node — the cached DataFrame itself, or a query
built on it — uses the same cache, keyed by (session_id, cache_id).
df.cache() → plan wrapped in BallistaCacheNode{cache_id, input} client, lazy
│ action, e.g. collect()
▼
lookup (session_id, cache_id); locations on live executors? scheduler
│
├─ MISS (cold) → build job runs `input` → pinned shuffle output
│ query job held (Queued; its id returned at once)
│ build done → record locations, pin the stage
│ │
└─ HIT (warm) ──────────────────────────┤
▼
cache node → scan of its shuffle output → ShuffleReaderExec,
a complete input of the stage reading it
▼
query job runs → results to the client
Cold path. The first action claims the cache and runs its subplan as its own build job, writing pinned shuffle output. The query becomes a second job, held without blocking until the build finishes; then it reads the output. The first action both builds and reads.
Warm path. A later action finds the cache built on live executors, so the cache node becomes a read and the subplan is never planned or run. A location on a dead executor turns the hit into a miss.
Both paths end the same way: the read is an ordinary
ShuffleReaderExec, registered as an input of the stage reading it (see The read job), so Ballista’s own rollback restores it like any other input. Losing the cache’s data mid-query fails the query cleanly for now; rebuild-and-resume is the next step (see Recovery).
A query waits on every build it needs, so independent caches build in parallel, a cache built on another builds after it, and a query that finds a cache already being built waits on that build. A build that fails or is cancelled fails the queries waiting on it. A cache that hits hides its subplan: caches nested inside it are not looked up. Before the held query runs, its caches are checked again, so one that went stale while it waited is rebuilt first.
The read job and external inputs#
Before the query job is planned, each cache node becomes a read with an external id:
logical Projection [c0 AS t.id, …] back to the cached names
TableScan ballista_cache_read_N [c0, …] scan of the cache's output
physical ShuffleReaderExec(stage_id = id, …) an ordinary shuffle read
graph stage.inputs[id] = locations, complete the read is a stage input
The scan is an ordinary TableScan whose provider returns the
ShuffleReaderExec (with a projection to the scan’s names), so every query
planner plans it and a session needs nothing installed to read caches. Its
columns are positional (c0, c1, …) because a table has one qualifier and
unique names, which a cached frame (a join, say) need not have; the projection
above restores the cached qualifiers and names, typed as the build job wrote
them.
The id comes from a reserved stage-id range (EXTERNAL_STAGE_ID_BASE, 2³⁰),
because a stage’s inputs are keyed by stage id: the read must not collide with
the job’s own stages, and two reads of one cache are two inputs. Ids are per job;
executors find data from the locations, not the id.
The entry is complete from the start, so it never delays the stage; it is there
so a rollback (for example, after the job loses one of its own shuffle outputs)
restores the reader, and so a lost cache can leave the stage waiting for a
rebuild. For each running job the scheduler also keeps a read record: per read,
the cache, the build read and the schema the build wrote, plus the subplan of a
read that can be rebuilt mid-query, to rebuild the cache if the job loses its
data (see Recovery). It holds nothing session-wide: a rebuild gets
a context made from the job’s configuration (JobState::new_context), and an
admission of its own.
Recovery#
When a running query loses cache data it reads (a fetch of it fails, or the scheduler loses an executor holding it), the stage reading it rolls back, the cache is rebuilt, and the stage resumes:
query job running; a stage reads cache c1
│ fetch failure, or an executor holding c1's data is lost
▼
stage rolls back (Ballista's own rollback — the read is a stage input)
▼
ExternalInputLost(job, id) no stage of this job can re-run c1's build
▼
read record (job, id) → c1, subplan, lost build
drop the lost build; rebuild c1 (or wait on a rebuild in flight)
▼
rebuild done → ExternalInputReady(job, id, new locations)
▼
stage resolves and re-runs → results to the client
Which data was lost. A failed fetch is cache data when it names one of the stage’s cache locations (executor, build stage, partition). The report carries no job id, so when one of the job’s own inputs on that executor matches too, both are recovered: the own producer re-runs its tasks there, and the cache is rebuilt. On executor loss, Ballista’s reset already drops the executor’s locations from every stage input, cache reads included, and rolls back their readers; the cache inputs it leaves incomplete are reported.
No producer re-run. For an own input the graph re-runs the producer’s lost tasks; a cache’s producer is another job, so the graph reports the input lost and the stage waits until it is provided again.
Shared rebuilds. Readers that lose the same cache share one rebuild, and one that finds a newer build resumes on it at once. A rebuild may have another partition count; the reader declares unknown partitioning, so the new partitions are folded into the old count. A rebuild must write the schema the read was planned with, as the read picks its columns by position; otherwise the job fails.
Retries and failure. Retries count towards
stage_max_failures, like any stage’s; a stage out of attempts fails the job with “lost DataFrame::cache() data”. A rebuild that fails, writes another schema, or is lost again before it is read fails the job with “could not recover a cache it reads: …”. The lost build was dropped before the rebuild started, so either way the retry rebuilds the cache.Admission. Rebuilds run off the event loop, and each is admitted on its own, even while the scheduler drains, as the job it serves already was. A scheduler shutting down waits for it until its build jobs are planned, like any submission, and then for them as running jobs.
Whole rebuild. The cache is rebuilt in full, which is correct whatever the subplan does; rebuilding only the lost partitions is a separate item (see Per-partition recovery). The reading stage re-runs in full; the job keeps its other stages.
A cache shared with another read is not rebuilt mid-query. A read depends on its cache and on every cache in its subplan, and its rebuild may rebuild any of them. If a subplan is not reproducible (
random(), aLIMITwithoutORDER BY, changed source files), a query whose lost read shares one of those caches with another of its reads (the same cache twice, one built from the other, or two built from a common one) would see two versions of it. It fails cleanly instead, and the retry reads one build throughout. A query reading only the outer cache of a nested chain still recovers: the rebuild brings back the inner caches it needs.The adaptive planner cannot yet re-resolve a stage it rolls back (for any job), so under it a job that loses cache data fails cleanly, and the retry rebuilds the cache.
A pipelined job (
ballista.execution.mode = pipelined) reads a cache built by an ordinary staged job. If the cache is lost before the job is placed, nothing has run yet, so the job waits for the rebuild and is placed reading it. Once placed, its tasks stream to each other and its results stream to the client, so restarting any of them would deliver rows twice; it fails cleanly instead, and the retry rebuilds the cache.
When lost cache data is rebuilt#
Found lost by |
What happens |
|---|---|
A new query reaching the cache |
Rebuilt before the query runs, whatever the planner or mode. |
A query held for its caches |
Linked to a new build and held again, up to |
A running query |
Rebuilt and resumed if the job is planned by the static planner (or is pipelined and not yet placed), its lost read shares no cache with another read, the stage has attempts left, and the rebuild succeeds with the same schema. Otherwise the job fails cleanly and the retry rebuilds. |
Registry and pinning#
(absent) --begin--> Pending{job} --complete(job)--> Materialized{job, schema, locations}
Pending{job} --abort(job)-----> (absent)
Materialized --invalidate------> (absent)
The registry is safe against the out-of-order events a distributed scheduler produces:
begin_materializationgives one job the claim; a concurrent miss getsAlreadyPendingand waits on that job (pending_job).complete_materializationpromotes only the claim owned by that job; anything else isStale, so a late completion cannot resurrect a removed entry or overwrite a newer claim.abort_materializationclears a failed job’s claim.invalidate_if(key, job)removes an entry only while it is stilljob’s, so acting on an entry seen earlier never drops a newer claim.Removals return the removed entries, so a caller can unpin their files.
The build job’s JobFinished handler records the entry before cleanup runs,
and cleanup skips the stages in pinned_stages(), so cleanup cannot race the
pin.
Invalidation#
Validate-on-read enforces “never read from a dead executor”: whenever a
cache is resolved into a read (when a query is submitted, and again before a held query runs), a location on an
executor that is not live turns the hit into a miss, and the dead entry is
dropped. A dropped entry’s build data is deleted after the usual delay for a
finished job’s data (finished_job_data_clean_up_interval_seconds), which lets
a query still reading it finish. Invalidation is whole-entry — a cache is rebuilt in full, never
served with missing partitions (see
Per-partition recovery).
invalidate_executor (on executor loss) and remove_session (on session
teardown) exist in the registry but are not called yet; until then a dead entry
is dropped when it is next read, or when a query loses its data.
Planning & configuration#
Planner. Nothing to install: a cache read is a table scan, which any session’s query planner plans, including a custom
override_session_builderand a scheduler created from a user’s state.Storage. Pinned shuffle output; for other media see Extension points.
Turning caching off.
ballista.cache.noop(a session setting, on by default) makescache()a no-op. An operator can turn caching off for the cluster with--disable-dataframe-cache(SchedulerConfig::disable_dataframe_cache): cache nodes then run inline.EXPLAINnever materializes: a built cache is shown as its read, any other as its subplan.EXPLAIN ANALYZEruns the query, so it materializes.
Durability & limits#
Scheduler restart. The in-memory registry loses its entries; caches are rebuilt on next use, and their old files stay until executor cleanup. A durable
CacheStatebackend could persist the entries.Ordering. A cache read declares no output ordering, so operators above a sorted cache may reorder it — as with DataFusion’s own
DataFrame::cache().Executor work-dir cleanup. If enabled (
job_data_clean_up_interval_seconds), an executor deletes job dirs untouched forjob_data_ttl_seconds, pinned caches included; the next read fails and drops the cache, and a retry rebuilds it.Mid-query loss. Under the static planner a query rebuilds lost cache data and carries on, unless the lost read shares a cache with another of its reads; otherwise it fails cleanly and the retry rebuilds (see Recovery).
Extension points#
Other storage media (e.g. memory with disk spill): the scheduler side deals only in
PartitionLocations, so a medium plugs in on the executors:override_execution_enginewrites the build stage into the store andoverride_arrow_flight_serviceserves it, whileShuffleReaderExecreads it back by location. Evicted data surfaces as a failed fetch, handled like lost cache data. Still needed then: a marker identifying the build stage, memory release on cleanup, and read locality (#2319) for zero-copy reads.CacheState: the registry backend; a durable or shared one would keep entries across scheduler restarts.External stage inputs: a stage input produced by another job is generic in the execution graph; a cache is the first producer, reusing another job’s shuffle output could be another.
Future work#
Eviction: entries leave today only when found dead, so a long-running scheduler accumulates cache data. Planned: a TTL / size-cap sweeper and idle-session expiry, using the removal-returns-entries contract.
Locality: place read tasks on the executor holding each partition (
TaskDistributionPolicy::Custom), Flight fetch as fallback (see #2319).
Per-partition recovery#
Ballista re-runs a lost map output per partition: a successful producer stage
re-runs only the tasks whose output was on the lost executor
(SuccessfulStage::reset_tasks, rerun_successful_stage), while a consumer
stage re-runs in full. The cache can follow the same split — the build side
rebuilds only the lost partitions, the reading stage still re-runs:
Materialized → Partial{missing} → partial build of missing partitions
→ merge locations → ExternalInputReady
Narrow subplans (no shuffle in the build’s final stage): restrict the subplan to the missing partitions (
restrict_plan_to_partitions).Subplans with a shuffle: the cleaned-up upstream stages re-run in full, unless intermediate outputs are pinned too.
Determinism: only plans that give partition
jthe same rows on every run qualify; anything else falls back to the whole rebuild.
Implementation plan#
Registry (done) —
CacheRegistry+CacheState/InMemoryCacheState.Materialize + read-back (done) — two jobs linked without blocking; builds on the two-job rewrite in #1993.
Pinning (done) — job-data cleanup honours
pinned_stages().Recovery, part 1 (done) — validate-on-read, re-checking a held query’s caches, and failing cleanly when a running query loses cache data.
Recovery, part 2 (done) — rebuild-and-resume for a running job under the static planner (Recovery).
Locality — place read tasks where partitions live (#2319).
Session teardown — wire
remove_sessionand sweep entries on executor loss (invalidate_executor), deleting their build data as invalidation does.Eviction — TTL / size cap / idle-session expiry.
Per-partition recovery — rebuild only lost partitions of narrow, deterministic subplans; later, pinned intermediates for subplans with a shuffle.