Extension author checklist#

The rules in this section, gathered into one list to run through before you publish. Each links to the page that explains it.

Protocol#

  • Every getter’s capsule name matches its method name. __datafusion_<thing>__ returns a capsule named datafusion_<thing>. → The capsule protocol

  • Your getter does not inspect its argument. Pass it to ffi_logical_codec_from_pycapsule and move on. It is not always a session, and when it is, it is not the Python SessionContext wrapper. → What your getter receives

  • You never construct a SessionContext inside your library. Take what the FFI constructors need off the argument you were handed. A context built inline is already dropped by the time the capsule is used. → What your getter receives

  • You do not depend on the datafusion-python crate. → Why FFI

Codecs#

  • Your codec claims narrowly. Downcast to your own types. Claiming a broad category takes nodes from every library installed after you and makes your library order-sensitive for everyone downstream. → When codec order matters

  • You declare __datafusion_codec_id__ if you might rename the class. The default id is the exporting class’s import path, so a rename stops older plans decoding. → Codec ids

  • Name-only decoders check name before trusting buf. An empty payload has no id to route on, so your try_decode_udf can be called with another library’s function name and an empty buffer. → Extension codecs

  • You round-trip a plan in a test and assert your codec did the work. Both being installed does not mean your node reached you. → When codec order matters

Bundles and planners#

  • You ship a bundle, not loose pieces, if you have codecs or a planner. → Extension bundles

  • Your bundle is configuration-only. Fresh components on every call, no cached bound components, no retaining the context passed in, no registering anything on it — a factory that mutates the context is not rolled back if a later factory raises. → Extension bundles

  • Your codecs are objects exposing the getter, not bare capsules. with_extensions refuses a capsule, because there would be nothing to name the codec by. → Codecs are objects, not capsules

  • Your planner hook wraps fallback and delegates to it. Ignoring it replaces every layer beneath you, which is legal but not composable. → Extension bundles

  • Your planner hook returns None, not fallback, when it has nothing to contribute. Returning fallback installs the session’s own planner as a foreign one and adds an FFI hop that was not there. → Extension bundles

  • You read the host’s codec chains in the planner hook, not the extension hook. Phase one runs before anything is installed. → Two phases, because codecs and planners compose differently

  • If you also offer the low-level path, document that codecs go in before a layered planner. → Install codecs before a layered planner

Packaging and documentation#

  • You state which datafusion version your release requires. A mismatch raises an ImportError on import, which is a good failure — but only if your users know what to install. → Mismatched extension libraries now fail loudly

  • You tell your users to keep a context alive for as long as anything derived from it is in use. This is the rule most likely to arrive as a bug report against your library. → Sessions, handles, and lifetimes

  • Your production codec serializes durable metadata, not a process-local token. The examples in this repository use tokens to make ownership observable; that is a demonstration, not a pattern. → Encode metadata, not a handle to a live object

  • You have integration tests across a real FFI boundary. The two example crates in this repository are the pattern: build the cdylib, install the wheel, then exercise it from Python.