How to use both dataset.select and selectExpr in apache spark

Sarvesh Belose picture Sarvesh Belose · Nov 22, 2017 · Viewed 13.6k times · Source

I want below mentioned data using Spark (2.2) dataset

Name    Age Age+5

A       10  15

B       5   10

C       25  30

I tried using the following :

dataset.select( 
        dataset.col("Name"), 
        dataset.col("Age),
        dataset.col( dataset.selectExpr("Age"+5).toString() )
       );

This throws exception as Age column not found.

Answer

philantrovert picture philantrovert · Nov 22, 2017

selectExpr has the definition :

public Dataset<Row> selectExpr(String... exprs)

It takes varargs String as it's parameter. So, you can just use :

dataset.selectExpr( "Name", "Age", "Age+5" )