How to Setup SPARK_HOME variable?

apache-spark h2o sparkling-water
roshan_ray picture roshan_ray · Oct 6, 2017 · Viewed 39.9k times · Source

Following the steps of Sparkling Water from the link http://h2o-release.s3.amazonaws.com/sparkling-water/rel-2.2/0/index.html.

Running in terminal :

~/InstallFile/SparklingWater/sparkling-water-2.2.0$ bin/sparkling-shell --conf "spark.executor.memory=1g"

Please setup SPARK_HOME variable to your Spark installation

Answer

Jader Martins picture Jader Martins · Oct 6, 2017

You should install and set the SPARK_HOME variable, in unix terminal run the following code to set the variable:

export SPARK_HOME="/path/to/spark"

To maintain this config you should append this to the end of your .bashrc.

See this for installation https://www.tutorialspoint.com/apache_spark/apache_spark_installation.htm

Related questions

How can I change column types in Spark SQL's DataFrame?

Suppose I'm doing something like: val df = sqlContext.load("com.databricks.spark.csv", Map("path" -> "cars.csv", "header" -> "true")) df.printSchema() root |-- year: string (nullable = true) |-- make: string (nullable = true) |-- model: string (nullable = …

Read More
Spark - load CSV file as DataFrame?

I would like to read a CSV in spark and convert it as DataFrame and store it in HDFS with df.registerTempTable("table_name") I have tried: scala> val df = sqlContext.load("hdfs:///csv/file/dir/file.csv") Error …

Read More
How to change dataframe column names in pyspark?

I come from pandas background and am used to reading data from CSV files into a dataframe and then simply changing the column names to something useful using the simple command: df.columns = new_column_name_list However, the same …

Read More