Apache Iceberg (experimental)#
The ballista-iceberg crate (integrations/iceberg) lets Ballista scan
Apache Iceberg tables. On top of
iceberg-datafusion, it adds the
codecs that send Iceberg tables and scans between the client, the scheduler and
the executors.
The crate is not published yet. It depends on unreleased revisions of
iceberg-datafusion and iceberg-rust, and needs Rust 1.95. It is its own
workspace, with its own Cargo.lock and CI, outside Ballista’s, and depends
on a pinned commit of Ballista rather than the crates beside it, so a
DataFusion upgrade in Ballista that iceberg-rust doesn’t support yet doesn’t
block Ballista’s build or release. Build and test it from
integrations/iceberg.
Iceberg support is experimental. Its behavior and API may change.
Usage#
Ready the session with register_iceberg, then mount an Iceberg catalog:
use std::sync::Arc;
use ballista::prelude::{SessionConfigExt, SessionContextExt};
use datafusion::execution::SessionStateBuilder;
use datafusion::prelude::{SessionConfig, SessionContext};
use ballista_iceberg::register_iceberg;
use iceberg_datafusion::IcebergCatalogProvider;
let state = SessionStateBuilder::new()
.with_config(SessionConfig::new_with_ballista())
.with_default_features()
.build();
let ctx = SessionContext::standalone_with_state(register_iceberg(state)).await?;
ctx.register_catalog("rest", Arc::new(IcebergCatalogProvider::try_new(catalog).await?));
ctx.sql("SELECT * FROM rest.ns.t").await?.show().await?;
A single table registered as an IcebergCatalogTableProvider works the same
way. Either kind reads the table’s latest version in each query.
For time travel, register an IcebergStaticTableProvider, which keeps one
version: IcebergStaticTableProvider::try_new_from_table_snapshot(table, snapshot_id).
The example registers both on a table in a temporary local warehouse, and shows a commit reaching the live table but not the static one:
cd integrations/iceberg
cargo run --example standalone-iceberg-scan
SQL clients#
Flight SQL clients (JDBC, ADBC, BI tools) send SQL to the scheduler, which plans
it, so they hold no catalog. To let them query Iceberg tables, mount catalogs
on the scheduler with SchedulerConfigExt::with_iceberg (the scheduler
feature), which also adds the Iceberg codecs:
use ballista_iceberg::SchedulerConfigExt;
let rest = IcebergCatalogProvider::try_new(catalog).await?;
let config = SchedulerConfig::default()
.with_flight_sql(true)
.with_iceberg([("rest", Arc::new(rest) as _)]);
Executors add the codecs with ExecutorProcessConfigExt::with_iceberg (the
executor feature):
use ballista_iceberg::ExecutorProcessConfigExt;
let config = ExecutorProcessConfig::default().with_iceberg();
Both keep any codecs the config already has, and the scheduler keeps its
session builder, mounting the catalogs on each session it builds. Call each
once, passing the scheduler every catalog: a second call adds the codecs
again, so the scheduler’s would no longer match the executors’. A client can
then run SELECT * FROM rest.ns.t. Tables are pinned the same way as on the
client, while the scheduler plans the query.
How it works#
Ballista sends the client’s plan to the scheduler, and each stage’s plan to
the executors, as bytes. The codecs replace each live Iceberg table with one
version of it: the location of its metadata file and its serialized FileIO.
Metadata files never change, so every node rebuilds exactly that version.
A catalog-backed table is pinned to the version its catalog serves when the client plans the query. DataFusion looks up each table a query names once, so every scan of it, such as each side of a self-join, reads that version, and each new query reads the latest one.
A static table is sent as the version it holds.
A scan is sent as the version the scheduler planned it against.
Only the location is sent, not the metadata, so a plan’s size doesn’t grow with the table’s history. Each node reads a version’s metadata file when it first rebuilds it, and keeps up to 16 rebuilt versions.
Only the node that plans a query talks to a catalog: the client, or the scheduler for SQL clients. Executors never need catalog configuration.
Combining with other table formats#
register_iceberg and with_iceberg add the Iceberg codecs after the codecs
already set, with ComposedLogicalExtensionCodec and DataFusion’s
ComposedPhysicalExtensionCodec (see
Combining Codecs). To read tables
of another format too, such as Delta, set its codecs first: on the session’s
SessionConfig before calling register_iceberg, and on SchedulerConfig and
ExecutorProcessConfig before calling with_iceberg. Clients, the scheduler
and executors must all set the same codecs in the same order.
Limitations#
Reads only.
INSERT INTOan Iceberg table fails with “isn’t supported on Ballista yet”.No metadata tables. Tables such as
t$snapshotscan’t be sent to the scheduler yet.One task per scan. Iceberg scans do not split into partitions yet, so one executor reads the whole table. Operators after the first shuffle still run in parallel. A scan reporting more than one partition fails to encode.
A
UNIONorUNION ALLcan fail or return wrong rows when the adaptive planner splits its stage between tasks. That happens withballista.scheduler.max_partitions_per_taskset, or when the executor has fewer free task slots than the union has partitions. A union of Iceberg scans fails with an error that suggestsSchedulerConfig::partition_sliceable_resolver, which standalone mode can’t set; #2852 fixes it. A union whose branches each merge their input into one partition, as with anORDER BYover the union, aLIMITin a branch or aggregates withoutGROUP BY, returns some rows more than once, whatever the table format (#2854). Settingballista.planner.adaptive.enabledtofalseavoids both.No encrypted tables. A table with
encryption.key-idset fails to load on the scheduler and executors, which have no key management client.Storage credentials travel in plain text. Each table’s serialized
FileIOincludes them, so the links between client, scheduler and executors must be trusted.A view keeps the version from when it was created. DataFusion stores a view as a planned query, and tables are pinned when a query is planned. The same goes for a
DataFrameor a Flight SQL prepared statement run more than once: each run reads the version from when it was planned. To read newer commits, recreate the view or plan the query again.A separately deployed scheduler and executors must link
ballista-icebergand be readied withwith_iceberg(see SQL clients). The tests cover standalone mode and an in-process scheduler serving Flight SQL.ctx.catalog(name)returns a wrapper.register_icebergwraps each catalog to pin its tables, so the result doesn’t downcast to the type you registered. Keep theArcyou registered to reach it.A mounted catalog lists its tables once.
IcebergCatalogProviderreads the table names when it is built, so a table created afterwards isn’t visible until the catalog is mounted again.