Top "Apache-spark-1.6" questions

In Spark 1.6.0 / Scala, is there an opportunity to get collect_list("colC") or collect_set("colC").over(Window.partitionBy("colA").…

Is there any configuration property we can set it to disable / enable Hive support through spark-shell explicitly in spark 1.6. I …

It's CDH with Spark 1.6. I am trying to import this Hypothetical CSV into a apache Spark DataFrame: $ hadoop fs -cat …

How can I get the first non-null values from a group by? I tried using first with coalesce F.first(…

I use Spark 1.6.1. We are trying to write an ORC file to HDFS using HiveContext and DataFrameWriter. While we can …

I am using spark version 1.6.3 and yarn version 2.7.1.2.3 comes with HDP-2.3.0.0-2557. Becuase, spark version is too old in the …

I use spark 1.6.1. My spark application reads more than 10000 parquet files stored in s3. val df = sqlContext.read.option("mergeSchema", "…

I am reading in a CSV as a Spark DataFrame and performing machine learning operations upon it. I keep getting …

I am working Spark v1.6. I have the following two DataFrames and I want to convert the null to 0 in …

I have a dataframe in PySpark which contains empty space, Null, and Nan. I want to remove rows which have …