Profiling Cookbook¶
The section contains examples how to perform CPU profiling for Apache DataFusion on different operating systems.
Building a flamegraph¶
Video: how to CPU profile DataFusion with a Flamegraph
A flamegraph is a visual representation of which functions are being run You can create flamegraphs in many ways; The instructions below are for cargo-flamegraph which results in images such as this:
MacOS¶
Step 1: Install the flamegraph Tool¶
To install flamegraph, run:
cargo install flamegraph
Step 2: Prepare Your Environment¶
Ensure that you’re in the directory containing the necessary data files for your DataFusion query. The flamegraph tool will profile the execution of your query against this data.
Step 3: Running the Flamegraph Tool¶
To generate a flamegraph, you’ll need to use the – separator to pass arguments to the binary you’re profiling. For datafusion-cli, you need to make sure to run the command with sudo permissions (especially on macOS, where DTrace requires elevated privileges).
Here is a general example:
sudo flamegraph -- datafusion-cli -f <path_to_sql_file/sql_file.sql>
Example: Generating a Flamegraph for a Specific Query¶
Here is an example using 28.sql:
sudo flamegraph -- datafusion-cli -f 28.sql
You can also invoke the flamegraph tool with cargo to profile a specific test or benchmark.
Example: Flamegraph for a specific test:¶
CARGO_PROFILE_RELEASE_DEBUG=true cargo flamegraph --root --unit-test datafusion -- dataframe::tests::test_array_agg
Example: Flamegraph for a benchmark¶
CARGO_PROFILE_RELEASE_DEBUG=true cargo flamegraph --root --bench sql_planner -- --bench