What's the equivalent of Panda's value_counts() in PySpark?

dataframe count pyspark pandas-groupby
TSAR picture TSAR · Jun 27, 2018 · Viewed 10.7k times · Source

I am having the following python/pandas command:

df.groupby('Column_Name').agg(lambda x: x.value_counts().max()

where I am getting the value counts for ALL columns in a DataFrameGroupBy object.

How do I do this action in PySpark?

Answer

Tanjin picture Tanjin · Jun 27, 2018

It's more or less the same:

spark_df.groupBy('column_name').count().orderBy('count')

In the groupBy you can have multiple columns delimited by a ,

For example groupBy('column_1', 'column_2')

Related questions

Count number of non-NaN entries in every column of Dataframe

I have a really big DataFrame and I was wondering if there was short (one or two liner) way to get the a count of non-NaN entries in a DataFrame. I don't want to do this one column at a …

Read More
Spark: How to translate count(distinct(value)) in Dataframe API's

I'm trying to compare different ways to aggregate my data. This is my input data with 2 elements (page,visitor): (PAG1,V1) (PAG1,V1) (PAG2,V1) (PAG2,V2) (PAG2,V1) (PAG1,V1) (PAG1,V2) (PAG1,V1) (PAG1,V2) (PAG1,V1) (PAG2,…

Read More
Python/Pandas: counting the number of missing/NaN in each row

I've got a dataset with a big number of rows. Some of the values are NaN, like this: In [91]: df Out[91]: 1 3 1 1 1 1 3 1 1 1 2 3 1 1 1 1 1 NaN NaN NaN 1 3 1 1 1 1 1 1 1 1 And I want to count the number of NaN values in each string, it would be …

Read More