Extension Guide

This section is for people writing a library that plugs into datafusion-python — a package that contributes table providers, functions, a catalog, extension codecs, or a query planner, usually written in Rust and exposed through PyO3.

Two neighbouring audiences are served elsewhere:

• Using an extension library someone else published: Using extension libraries.

• Changing datafusion-python itself, including the framing that makes this protocol work: Contributor Guide.

The protocol described here is a public, versioned contract. When a hook’s signature changes, the change is documented in Upgrade Guides with a before-and-after, and objects built against a different DataFusion major version are rejected with a clear error rather than used as-is.

Start here

If you have not built an extension before, read Why FFI and The capsule protocol in order. Together they explain why your library must not depend on the datafusion-python crate, and the one convention every hook follows. After that the pages are independent — go to the one matching what you are contributing.

The three roles in a query

A single query can involve three independent native libraries, and much of this section only makes sense once they are distinct in your head:

• datafusion-python — the host. Owns the session, and decodes whatever comes back from the other two.

• A provider library — owns table providers, catalogs, and functions, plus the codecs that serialize them.

• A planner library — owns a query planner and the configuration it needs.

The worked examples in this repository use two separate crates, datafusion-ffi-example and datafusion-ffi-query-planner-example, so each role has a distinct shared-library identity. A real library may play more than one role; keeping them separate in the examples is what makes the boundaries observable.

The session owns the codecs used for the exchange and supplies them to the foreign planner. That is what lets the planner decode provider-owned objects, and lets datafusion-python decode the physical plan the planner returns.

Hook reference

Every integration point is a dunder method named __datafusion_*__. The convention is uniform enough to be worth stating once: your object exposes the getter, datafusion-python calls it, and it returns a PyCapsule wrapping an FFI-safe struct. See The capsule protocol for what that means and What your getter receives for the argument every getter in the middle group receives.

Each hook’s capsule name follows from its own name by the naming rule, so it is not repeated here.

Hook	Contributes	Argument	Documented on
__datafusion_table_provider__	one table	codec source	Providers and catalogs
__datafusion_table_provider_factory__	tables built by CREATE EXTERNAL TABLE	codec source	Providers and catalogs
__datafusion_catalog_provider__	a named set of schemas	codec source	Providers and catalogs
__datafusion_catalog_provider_list__	the whole catalog namespace	codec source	Providers and catalogs
__datafusion_schema_provider__	a named set of tables	codec source	Providers and catalogs
__datafusion_table_function__	a table-valued function	session	Functions and table functions
__datafusion_scalar_udf__	a scalar function	none	Functions and table functions
__datafusion_aggregate_udf__	an aggregate function	none	Functions and table functions
__datafusion_window_udf__	a window function	none	Functions and table functions
__datafusion_logical_extension_codec__	a logical codec	session	Extension codecs
__datafusion_physical_extension_codec__	a physical codec	session	Extension codecs
__datafusion_codec_id__	the wire id a codec’s payloads carry	—	Codec ids
__datafusion_query_planner__	a query planner	session	Query planners
__datafusion_session_components__	a bundle’s codecs	ctx	Extension bundles
__datafusion_session_planner__	a bundle’s planner, wrapping fallback	ctx, fallback	Extension bundles
__datafusion_physical_optimizer_rule__	a physical optimizer rule	none	Optimizer rules and configuration
__datafusion_extension_options__	typed entries in the session config	none	Optimizer rules and configuration
__datafusion_task_context_provider__	the host’s task context	none	__datafusion_task_context_provider__

Three rows are not like the others. __datafusion_task_context_provider__ is implemented by the host, not by your library — you read it off the session you are handed. __datafusion_codec_id__ is a plain string attribute rather than a method returning a capsule. And the two session_ hooks are dispatched from Python and return objects rather than capsules.

“codec source” in the argument column means the value is something you can read the host’s logical extension codec off, which is not always a session. What your getter receives explains why, and what to do with it.

python/tests/test_docstrings.py compares this table against the hook names the package actually dispatches, so a hook added or renamed without touching this page fails the suite.

• Why FFI
  • The primary issue
  • The FFI approach
  • Inspiration from Arrow
  • If FFI does not yet cover what you need
• The capsule protocol
  • FFI-safe types
  • FFI_ on the provider, Foreign on the receiver
  • Wrapping it in a capsule
  • The naming rule
  • Version checking
  • What your getter receives
  • __datafusion_task_context_provider__
• Providers and catalogs
  • A table provider
  • The catalog family
  • A table provider factory
  • Serializing what you expose
• Functions and table functions
  • The three scalar-shaped hooks
  • Table functions
  • Serializing functions
• Extension codecs
  • Codecs compose
  • Encode metadata, not a handle to a live object
  • Codec ids
  • Functions whose name is the whole encoding
  • When codec order matters
  • A codec decodes against the session running the query
• Extension bundles
  • Why not have callers install the pieces
  • A bundle you can run
  • Two phases, because codecs and planners compose differently
  • Returning a planner, or not
  • When the two orders conflict
  • Codecs are objects, not capsules
  • Failure and rollback
• Query planners
  • One planner per session
  • Install codecs before a layered planner
• Optimizer rules and configuration
  • A physical optimizer rule
  • Typed configuration
• Sessions, handles, and lifetimes
  • The rule that surprises people
  • What a derived context shares
  • Keep a context alive
• Extension author checklist
  • Protocol
  • Codecs
  • Bundles and planners
  • Packaging and documentation