datafusion.catalog¶

Data catalog providers.

Classes¶

`Catalog`	DataFusion data catalog.
`CatalogList`	DataFusion data catalog list.
`CatalogProvider`	Abstract class for defining a Python based Catalog Provider.
`CatalogProviderList`	Abstract class for defining a Python based Catalog Provider List.
`Schema`	DataFusion Schema.
`SchemaProvider`	Abstract class for defining a Python based Schema Provider.
`Table`	A DataFusion table.

Module Contents¶

class datafusion.catalog.Catalog(catalog: datafusion._internal.catalog.RawCatalog)¶

DataFusion data catalog.

This constructor is not typically called by the end user.

__repr__() → str¶: Print a string representation of the catalog.

database(name: str = 'public') → Schema¶: Returns the database with the given name from this catalog.

deregister_schema(name: str, cascade: bool = True) → Schema | None¶: Deregister a schema from this catalog.

static memory_catalog(ctx: datafusion.SessionContext | None = None) → Catalog¶: Create an in-memory catalog provider.

names() → set[str]¶: This is an alias for schema_names.

register_schema(name: str, schema: Schema | SchemaProvider | SchemaProviderExportable) → Schema | None¶: Register a schema with this catalog.

schema(name: str = 'public') → Schema¶: Returns the database with the given name from this catalog.

schema_names() → set[str]¶: Returns the list of schemas in this catalog.

catalog¶

class datafusion.catalog.CatalogList(catalog_list: datafusion._internal.catalog.RawCatalogList)¶

DataFusion data catalog list.

This constructor is not typically called by the end user.

__repr__() → str¶: Print a string representation of the catalog list.

catalog(name: str = 'datafusion') → Catalog¶: Returns the catalog with the given name from this catalog.

catalog_names() → set[str]¶: Returns the list of schemas in this catalog.

static memory_catalog(ctx: datafusion.SessionContext | None = None) → CatalogList¶: Create an in-memory catalog provider list.

names() → set[str]¶: This is an alias for catalog_names.

register_catalog(name: str, catalog: Catalog | CatalogProvider | CatalogProviderExportable) → Catalog | None¶: Register a catalog with this catalog list.

catalog_list¶

class datafusion.catalog.CatalogProvider¶

Bases: abc.ABC

Abstract class for defining a Python based Catalog Provider.

deregister_schema(name: str, cascade: bool) → None¶

Remove a schema from this catalog.

This method is optional. If your catalog provides a fixed list of schemas, you do not need to implement this method.

Parameters:

name – The name of the schema to remove.
cascade – If true, deregister the tables within the schema.

register_schema(name: str, schema: SchemaProviderExportable | SchemaProvider | Schema) → None¶

Add a schema to this catalog.

This method is optional. If your catalog provides a fixed list of schemas, you do not need to implement this method.

abstract schema(name: str) → Schema | None¶: Retrieve a specific schema from this catalog.

abstract schema_names() → set[str]¶: Set of the names of all schemas in this catalog.

class datafusion.catalog.CatalogProviderList¶

Bases: abc.ABC

Abstract class for defining a Python based Catalog Provider List.

abstract catalog(name: str) → CatalogProviderExportable | CatalogProvider | Catalog | None¶: Retrieve a specific catalog from this catalog list.

abstract catalog_names() → set[str]¶: Set of the names of all catalogs in this catalog list.

register_catalog(name: str, catalog: CatalogProviderExportable | CatalogProvider | Catalog) → None¶

Add a catalog to this catalog list.

This method is optional. If your catalog provides a fixed list of catalogs, you do not need to implement this method.

class datafusion.catalog.Schema(schema: datafusion._internal.catalog.RawSchema)¶

DataFusion Schema.

This constructor is not typically called by the end user.

__repr__() → str¶: Print a string representation of the schema.

deregister_table(name: str) → None¶: Deregister a table provider from this schema.

static memory_schema(ctx: datafusion.SessionContext | None = None) → Schema¶: Create an in-memory schema provider.

names() → set[str]¶: This is an alias for table_names.

register_table(name: str, table: Table | datafusion.context.TableProviderExportable | datafusion.DataFrame | pyarrow.dataset.Dataset) → None¶: Register a table in this schema.

table(name: str) → Table¶: Return the table with the given name from this schema.

table_exist(name: str) → bool¶: Determines if a table exists in this schema.

table_names() → set[str]¶: Returns the list of all tables in this schema.

_raw_schema¶

class datafusion.catalog.SchemaProvider¶

Bases: abc.ABC

Abstract class for defining a Python based Schema Provider.

deregister_table(name: str, cascade: bool) → None¶

Remove a table from this schema.

This method is optional. If your schema provides a fixed list of tables, you do not need to implement this method.

owner_name() → str | None¶

Returns the owner of the schema.

This is an optional method. The default return is None.

register_table(name: str, table: Table | datafusion.context.TableProviderExportable | Any) → None¶

Add a table to this schema.

This method is optional. If your schema provides a fixed list of tables, you do not need to implement this method.

abstract table(name: str) → Table | None¶: Retrieve a specific table from this schema.

abstract table_exist(name: str) → bool¶: Returns true if the table exists in this schema.

abstract table_names() → set[str]¶: Set of the names of all tables in this schema.

class datafusion.catalog.Table(table: Table | datafusion.context.TableProviderExportable | datafusion.DataFrame | pyarrow.dataset.Dataset, ctx: datafusion.SessionContext | None = None)¶

A DataFusion table.

Internally we currently support the following types of tables:

Tables created using built-in DataFusion methods, such as reading from CSV or Parquet
pyarrow datasets
DataFusion DataFrames, which will be converted into a view
Externally provided tables implemented with the FFI PyCapsule interface (advanced)

Constructor.

__repr__() → str¶: Print a string representation of the table.

static from_dataset(dataset: pyarrow.dataset.Dataset) → Table¶: Turn a pyarrow.dataset Dataset into a Table.

__slots__ = ('_inner',)¶

_inner¶

property kind: str¶: Returns the kind of table.

property schema: pyarrow.Schema¶: Returns the schema associated with this table.

datafusion

datafusion.context