Apache Iceberg (experimental)#

The ballista-iceberg crate (integrations/iceberg) lets Ballista scan Apache Iceberg tables. On top of iceberg-datafusion, it adds the codecs that send Iceberg tables and scans between the client, the scheduler and the executors.

The crate is not published yet. It depends on unreleased revisions of iceberg-datafusion and iceberg-rust, and needs Rust 1.95. It is its own workspace, with its own Cargo.lock and CI, outside Ballista’s, and depends on a pinned commit of Ballista rather than the crates beside it, so a DataFusion upgrade in Ballista that iceberg-rust doesn’t support yet doesn’t block Ballista’s build or release. Build and test it from integrations/iceberg.

Iceberg support is experimental. Its behavior and API may change.

Usage#

Ready the session with register_iceberg, then mount an Iceberg catalog:

use std::sync::Arc;

use ballista::prelude::{SessionConfigExt, SessionContextExt};
use datafusion::execution::SessionStateBuilder;
use datafusion::prelude::{SessionConfig, SessionContext};
use ballista_iceberg::register_iceberg;
use iceberg_datafusion::IcebergCatalogProvider;

let state = SessionStateBuilder::new()
    .with_config(SessionConfig::new_with_ballista())
    .with_default_features()
    .build();
let ctx = SessionContext::standalone_with_state(register_iceberg(state)).await?;

ctx.register_catalog("rest", Arc::new(IcebergCatalogProvider::try_new(catalog).await?));
ctx.sql("SELECT * FROM rest.ns.t").await?.show().await?;

A single table registered as an IcebergCatalogTableProvider works the same way. Either kind reads the table’s latest version in each query.

For time travel, register an IcebergStaticTableProvider, which keeps one version: IcebergStaticTableProvider::try_new_from_table_snapshot(table, snapshot_id).

The example registers both on a table in a temporary local warehouse, and shows a commit reaching the live table but not the static one:

cd integrations/iceberg
cargo run --example standalone-iceberg-scan

SQL clients#

Flight SQL clients (JDBC, ADBC, BI tools) send SQL to the scheduler, which plans it, so they hold no catalog. To let them query Iceberg tables, mount catalogs on the scheduler with SchedulerConfigExt::with_iceberg (the scheduler feature), which also adds the Iceberg codecs:

use ballista_iceberg::SchedulerConfigExt;

let rest = IcebergCatalogProvider::try_new(catalog).await?;
let config = SchedulerConfig::default()
    .with_flight_sql(true)
    .with_iceberg([("rest", Arc::new(rest) as _)]);

Executors add the codecs with ExecutorProcessConfigExt::with_iceberg (the executor feature):

use ballista_iceberg::ExecutorProcessConfigExt;

let config = ExecutorProcessConfig::default().with_iceberg();

Both keep any codecs the config already has, and the scheduler keeps its session builder, mounting the catalogs on each session it builds. Call each once, passing the scheduler every catalog: a second call adds the codecs again, so the scheduler’s would no longer match the executors’. A client can then run SELECT * FROM rest.ns.t. Tables are pinned the same way as on the client, while the scheduler plans the query.

How it works#

Ballista sends the client’s plan to the scheduler, and each stage’s plan to the executors, as bytes. The codecs replace each live Iceberg table with one version of it: the location of its metadata file and its serialized FileIO. Metadata files never change, so every node rebuilds exactly that version.

  • A catalog-backed table is pinned to the version its catalog serves when the client plans the query. DataFusion looks up each table a query names once, so every scan of it, such as each side of a self-join, reads that version, and each new query reads the latest one.

  • A static table is sent as the version it holds.

  • A scan is sent as the version the scheduler planned it against.

Only the location is sent, not the metadata, so a plan’s size doesn’t grow with the table’s history. Each node reads a version’s metadata file when it first rebuilds it, and keeps up to 16 rebuilt versions.

Only the node that plans a query talks to a catalog: the client, or the scheduler for SQL clients. Executors never need catalog configuration.

Combining with other table formats#

register_iceberg and with_iceberg add the Iceberg codecs after the codecs already set, with ComposedLogicalExtensionCodec and DataFusion’s ComposedPhysicalExtensionCodec (see Combining Codecs). To read tables of another format too, such as Delta, set its codecs first: on the session’s SessionConfig before calling register_iceberg, and on SchedulerConfig and ExecutorProcessConfig before calling with_iceberg. Clients, the scheduler and executors must all set the same codecs in the same order.

Limitations#

  • Reads only. INSERT INTO an Iceberg table fails with “isn’t supported on Ballista yet”.

  • No metadata tables. Tables such as t$snapshots can’t be sent to the scheduler yet.

  • One task per scan. Iceberg scans do not split into partitions yet, so one executor reads the whole table. Operators after the first shuffle still run in parallel. A scan reporting more than one partition fails to encode.

  • A UNION or UNION ALL can fail or return wrong rows when the adaptive planner splits its stage between tasks. That happens with ballista.scheduler.max_partitions_per_task set, or when the executor has fewer free task slots than the union has partitions. A union of Iceberg scans fails with an error that suggests SchedulerConfig::partition_sliceable_resolver, which standalone mode can’t set; #2852 fixes it. A union whose branches each merge their input into one partition, as with an ORDER BY over the union, a LIMIT in a branch or aggregates without GROUP BY, returns some rows more than once, whatever the table format (#2854). Setting ballista.planner.adaptive.enabled to false avoids both.

  • No encrypted tables. A table with encryption.key-id set fails to load on the scheduler and executors, which have no key management client.

  • Storage credentials travel in plain text. Each table’s serialized FileIO includes them, so the links between client, scheduler and executors must be trusted.

  • A view keeps the version from when it was created. DataFusion stores a view as a planned query, and tables are pinned when a query is planned. The same goes for a DataFrame or a Flight SQL prepared statement run more than once: each run reads the version from when it was planned. To read newer commits, recreate the view or plan the query again.

  • A separately deployed scheduler and executors must link ballista-iceberg and be readied with with_iceberg (see SQL clients). The tests cover standalone mode and an in-process scheduler serving Flight SQL.

  • ctx.catalog(name) returns a wrapper. register_iceberg wraps each catalog to pin its tables, so the result doesn’t downcast to the type you registered. Keep the Arc you registered to reach it.

  • A mounted catalog lists its tables once. IcebergCatalogProvider reads the table names when it is built, so a table created afterwards isn’t visible until the catalog is mounted again.