PySpark 2.0 The size or shape of a DataFrame

dataframe size pyspark shape
Xi Liang picture Xi Liang · Sep 23, 2016 · Viewed 153.8k times · Source

I am trying to find out the size/shape of a DataFrame in PySpark. I do not see a single function that can do this.

In Python I can do

data.shape()

Is there a similar function in PySpark. This is my current solution, but I am looking for an element one

row_number = data.count()
column_number = len(data.dtypes)

The computation of the number of columns is not ideal...

Answer

George Fisher picture George Fisher · Aug 11, 2017

You can get its shape with:

print((df.count(), len(df.columns)))

Related questions

How to create a DataFrame of random integers with Pandas?

I know that if I use randn, import pandas as pd import numpy as np df = pd.DataFrame(np.random.randn(100, 4), columns=list('ABCD')) gives me what I am looking for, but with elements from a normal distribution. But what …

Read More
How to select rows from a DataFrame based on column values

How can I select rows from a DataFrame based on values in some column in Pandas? In SQL, I would use: SELECT * FROM table WHERE colume_name = some_value I tried to look at Pandas' documentation, but I did not …

Read More
Renaming columns in pandas

I have a DataFrame using pandas and column labels that I need to edit to replace the original column labels. I'd like to change the column names in a DataFrame A where the original column names are: ['$a', '$…

Read More