Optimizer rules and configuration

Two hooks contribute things that are neither data nor functions: a rewrite pass over physical plans, and typed entries in the session config. Both take no argument and both are implemented in datafusion-ffi-example.

A physical optimizer rule

__datafusion_physical_optimizer_rule__ contributes a rule that rewrites physical plans, installed with add_physical_optimizer_rule(). Reach for this rather than a query planner when you want to adjust the plan DataFusion produced rather than produce it yourself — it is much the smaller commitment, and rules accumulate where planners nest.

fn __datafusion_physical_optimizer_rule__<'py>(
    &self,
    py: Python<'py>,
) -> PyResult<Bound<'py, PyCapsule>> {
    let rule: Arc<dyn PhysicalOptimizerRule + Send + Sync> = Arc::new(self.clone());
    let runtime = get_tokio_runtime().handle().clone();
    let ffi = FFI_PhysicalOptimizerRule::new(rule, Some(runtime));

    PyCapsule::new_with_value(py, ffi, cr"datafusion_physical_optimizer_rule")
}

Typed configuration

__datafusion_extension_options__ contributes typed configuration entries that your components can read back out of the session config, installed with SessionConfig.with_extension. FFI_ExtensionOptions carries no version field, so it is one of the three components that cannot be version-checked on import.

fn __datafusion_extension_options__<'py>(
    &self,
    py: Python<'py>,
) -> PyResult<Bound<'py, PyCapsule>> {
    let mut config = FFI_ExtensionOptions::default();
    config
        .add_config(self)
        .map_err(|e| PyRuntimeError::new_err(e.to_string()))?;

    PyCapsule::new_with_value(py, config, cr"datafusion_extension_options")
}