DataFrames¶

Overview¶

DataFusion’s DataFrame API provides a powerful interface for building and executing queries against data sources. It offers a familiar API similar to pandas and other DataFrame libraries, but with the performance benefits of Rust and Arrow.

A DataFrame represents a logical plan that can be composed through operations like filtering, projection, and aggregation. The actual execution happens when terminal operations like collect() or show() are called.

Basic Usage¶

import datafusion
from datafusion import col, lit

# Create a context and register a data source
ctx = datafusion.SessionContext()
ctx.register_csv("my_table", "path/to/data.csv")

# Create and manipulate a DataFrame
df = ctx.sql("SELECT * FROM my_table")

# Or use the DataFrame API directly
df = (ctx.table("my_table")
      .filter(col("age") > lit(25))
      .select([col("name"), col("age")]))

# Execute and collect results
result = df.collect()

# Display the first few rows
df.show()

HTML Rendering¶

When working in Jupyter notebooks or other environments that support HTML rendering, DataFrames will automatically display as formatted HTML tables, making it easier to visualize your data.

The _repr_html_ method is called automatically by Jupyter to render a DataFrame. This method controls how DataFrames appear in notebook environments, providing a richer visualization than plain text output.

Customizing HTML Rendering¶

You can customize how DataFrames are rendered in HTML by configuring the formatter:

from datafusion.html_formatter import configure_formatter

# Change the default styling
configure_formatter(
    max_cell_length=25,        # Maximum characters in a cell before truncation
    max_width=1000,            # Maximum width in pixels
    max_height=300,            # Maximum height in pixels
    max_memory_bytes=2097152,  # Maximum memory for rendering (2MB)
    min_rows_display=20,       # Minimum number of rows to display
    repr_rows=10,              # Number of rows to display in __repr__
    enable_cell_expansion=True,# Allow expanding truncated cells
    custom_css=None,           # Additional custom CSS
    show_truncation_message=True, # Show message when data is truncated
    style_provider=None,       # Custom styling provider
    use_shared_styles=True     # Share styles across tables
)

The formatter settings affect all DataFrames displayed after configuration.

Custom Style Providers¶

For advanced styling needs, you can create a custom style provider:

from datafusion.html_formatter import StyleProvider, configure_formatter

class MyStyleProvider(StyleProvider):
    def get_table_styles(self):
        return {
            "table": "border-collapse: collapse; width: 100%;",
            "th": "background-color: #007bff; color: white; padding: 8px; text-align: left;",
            "td": "border: 1px solid #ddd; padding: 8px;",
            "tr:nth-child(even)": "background-color: #f2f2f2;",
        }

    def get_value_styles(self, dtype, value):
        """Return custom styles for specific values"""
        if dtype == "float" and value < 0:
            return "color: red;"
        return None

# Apply the custom style provider
configure_formatter(style_provider=MyStyleProvider())

Performance Optimization with Shared Styles¶

The use_shared_styles parameter (enabled by default) optimizes performance when displaying multiple DataFrames in notebook environments:

When use_shared_styles=True: - CSS styles and JavaScript are included only once per notebook session - This reduces HTML output size and prevents style duplication - Improves rendering performance with many DataFrames - Applies consistent styling across all DataFrames

Creating a Custom Formatter¶

For complete control over rendering, you can implement a custom formatter:

from datafusion.html_formatter import Formatter, get_formatter

class MyFormatter(Formatter):
    def format_html(self, batches, schema, has_more=False, table_uuid=None):
        # Create your custom HTML here
        html = "<div class='my-custom-table'>"
        # ... formatting logic ...
        html += "</div>"
        return html

# Set as the global formatter
configure_formatter(formatter_class=MyFormatter)

# Or use the formatter just for specific operations
formatter = get_formatter()
custom_html = formatter.format_html(batches, schema)

Managing Formatters¶

Reset to default formatting:

from datafusion.html_formatter import reset_formatter

# Reset to default settings
reset_formatter()

Get the current formatter settings:

from datafusion.html_formatter import get_formatter

formatter = get_formatter()
print(formatter.max_rows)
print(formatter.theme)

Contextual Formatting¶

You can also use a context manager to temporarily change formatting settings:

from datafusion.html_formatter import formatting_context

# Default formatting
df.show()

# Temporarily use different formatting
with formatting_context(max_rows=100, theme="dark"):
    df.show()  # Will use the temporary settings

# Back to default formatting
df.show()

Memory and Display Controls¶

You can control how much data is displayed and how much memory is used for rendering:

configure_formatter(
    max_memory_bytes=4 * 1024 * 1024,  # 4MB maximum memory for display
    min_rows_display=50,               # Always show at least 50 rows
    repr_rows=20                       # Show 20 rows in __repr__ output
)

These parameters help balance comprehensive data display against performance considerations.

Data Sources

Common Operations