Custom Table Provider

If you have a custom data source that you want to integrate with DataFusion, you can do so by implementing the TableProvider interface in Rust and then exposing it in Python. To do so, you must use DataFusion 43.0.0 or later and expose a FFI_TableProvider via PyCapsule.

A complete example can be found in the examples folder. For how to write one — the getter, what it receives, and how to serialize what it exposes — see Providers and catalogs in the Extension Guide.

Once you have this library available, you can construct a Table in Python and register it with the SessionContext.

from datafusion import SessionContext, Table

ctx = SessionContext()
provider = MyTableProvider()

ctx.register_table("capsule_table", provider)

ctx.table("capsule_table").show()