In Spark SQL when I tried to use map function on DataFrame then I am getting below error.
The method map(Function1, ClassTag) in the type DataFrame is not applicable for the arguments (new Function(){})
I am following spark 1.3 documentation as well. https://spark.apache.org/docs/latest/sql-programming-guide.html#inferring-the-schema-using-reflection Have any one solution?
Here is my testing code.
// SQL can be run over RDDs that have been registered as tables.
DataFrame teenagers = sqlContext.sql("SELECT name FROM people WHERE age >= 13 AND age <= 19");
List<String> teenagerNames = teenagers.map(
new Function<Row, String>() {
public String call(Row row) {
return "Name: " + row.getString(0);
}
}).collect();
Change this to:
Java 6 & 7
List<String> teenagerNames = teenagers.javaRDD().map(
new Function<Row, String>() {
public String call(Row row) {
return "Name: " + row.getString(0);
}
}).collect();
Java 8
List<String> t2 = teenagers.javaRDD().map(
row -> "Name: " + row.getString(0)
).collect();
Once you call javaRDD() it works just like any other RDD map function.
This works with Spark 1.3.0 and up.