Query planners#

A query planner turns a logical plan into a physical one. Contributing your own is how a library changes the way queries execute rather than what data they can reach — a distributed engine is the motivating case, and an optimizing rewriter is another.

The session owns the codecs used for the exchange and supplies them to your planner. That is what lets the planner decode provider-owned objects, and lets datafusion-python decode the physical plan the planner returns. Your planner uses FFI_QueryPlanner::new_with_ffi_codecs with the two codecs it takes off the session, and never touches a task-context provider directly. That also matches what installation does anyway: set_query_planner() builds the planner against the codecs of the session that will run the query.

MyQueryPlanner in datafusion-ffi-query-planner-example is the worked implementation.

One planner per session#

A session holds exactly one query planner. Calling set_query_planner again replaces the installed planner instead of layering another one.

To chain planners, have the new planner wrap the capsule returned by SessionContext.__datafusion_query_planner__, captured before the new planner is installed, and delegate to it explicitly:

fallback = ctx.__datafusion_query_planner__()
ctx.set_query_planner(MyPlanner(fallback=fallback))

set_query_planner returns nothing. The query planner lives in SessionState, so it is a property of the session rather than of a handle on it, and installing one is visible to every context sharing that session. See Sessions, handles, and lifetimes.

If the planners you are composing all ship as bundles, prefer with_extensions — it does the nesting for you and cannot capture a partial codec chain.

Install codecs before a layered planner#

Installing a codec on a session that already has a foreign planner rebuilds that planner against the new chain: there is one planner, and it has to carry the codecs currently in force. But that rebuild swaps the codecs on the installed ForeignQueryPlanner handle, and only that handle.

A planner that wraps a fallback resolved that fallback when it was installed, and holds the result inside its own library’s private data — behind a create_physical_plan function pointer, with no Python-side handle. A codec installed afterwards therefore reaches the outer planner and not the fallback, which keeps whichever codecs were in force when it was imported.

The stale codecs stay usable rather than dangling — they hold weak handles to the one Arc<SessionContext> the session keeps alive — so the effect is a fallback hop serializing with an older codec, not a failure. Neither side can repair it; the reasons are in Why planner codec rebinding is one level deep, and a fix has to come from upstream (apache/datafusion#24762).

So, three rules:

  • Install the codecs before a layered planner.

  • If a codec has to go in afterwards, install the outer planner again on the handle that holds the new codec — that re-runs its getter, which re-imports the fallback against that handle’s codecs. Re-installing on the original handle rebinds the session’s planner back to the original handle’s codecs instead, which is the trap the query-planner example’s test suite pins.

  • Better, use Extension bundles, where there is no “afterwards” within a call.

Note

This is invisible in the examples here, which use one fallback in the same cdylib as its wrapper; datafusion-ffi short-circuits a same-library hop rather than serializing, so no codec runs. A fallback in a different library would serialize, and would do it with the codecs it was imported with.