Basic Operations#
In this section, you will learn how to display essential details of DataFrames using specific functions.
from datafusion import SessionContext
import random
ctx = SessionContext()
df = ctx.from_pydict({
"nrs": [1, 2, 3, 4, 5],
"names": ["python", "ruby", "java", "haskell", "go"],
"random": random.sample(range(1000), 5),
"groups": ["A", "A", "B", "C", "B"],
})
df
DataFrame()
+-----+---------+--------+--------+
| nrs | names | random | groups |
+-----+---------+--------+--------+
| 1 | python | 612 | A |
| 2 | ruby | 819 | A |
| 3 | java | 198 | B |
| 4 | haskell | 247 | C |
| 5 | go | 457 | B |
+-----+---------+--------+--------+
Use limit() to view the top rows of the frame:
df.limit(2)
DataFrame()
+-----+--------+--------+--------+
| nrs | names | random | groups |
+-----+--------+--------+--------+
| 1 | python | 612 | A |
| 2 | ruby | 819 | A |
+-----+--------+--------+--------+
Display the columns of the DataFrame using schema():
df.schema()
nrs: int64
names: string
random: int64
groups: string
The method to_pandas() uses pyarrow to convert to pandas DataFrame, by collecting the batches,
passing them to an Arrow table, and then converting them to a pandas DataFrame.
df.to_pandas()
nrs names random groups
0 1 python 612 A
1 2 ruby 819 A
2 3 java 198 B
3 4 haskell 247 C
4 5 go 457 B
describe() shows a quick statistic summary of your data:
df.describe()
DataFrame()
+------------+--------------------+-------+--------------------+--------+
| describe | nrs | names | random | groups |
+------------+--------------------+-------+--------------------+--------+
| count | 5.0 | 5 | 5.0 | 5 |
| null_count | 0.0 | 0 | 0.0 | 0 |
| mean | 3.0 | null | 466.6 | null |
| std | 1.5811388300841898 | null | 257.77373799516505 | null |
| min | 1.0 | go | 198.0 | A |
| max | 5.0 | ruby | 819.0 | C |
| median | 3.0 | null | 457.0 | null |
+------------+--------------------+-------+--------------------+--------+