Spark : how to run spark file from spark shell

scala apache-spark cloudera-cdh cloudera-manager
Ramakrishna picture Ramakrishna · Dec 31, 2014 · Viewed 106.3k times · Source


I am using CDH 5.2. I am able to use spark-shell to run the commands.

  1. How can I run the file(file.spark) which contain spark commands.
  2. Is there any way to run/compile the scala programs in CDH 5.2 without sbt?

Thanks in advance

Answer

Ziyao Li picture Ziyao Li · Mar 17, 2015

In command line, you can use 

spark-shell -i file.scala

to run code which is written in file.scala

Related questions

How can I change column types in Spark SQL's DataFrame?

Suppose I'm doing something like: val df = sqlContext.load("com.databricks.spark.csv", Map("path" -> "cars.csv", "header" -> "true")) df.printSchema() root |-- year: string (nullable = true) |-- make: string (nullable = true) |-- model: string (nullable = …

Read More
Spark - load CSV file as DataFrame?

I would like to read a CSV in spark and convert it as DataFrame and store it in HDFS with df.registerTempTable("table_name") I have tried: scala> val df = sqlContext.load("hdfs:///csv/file/dir/file.csv") Error …

Read More
How to convert rdd object to dataframe in spark

How can I convert an RDD (org.apache.spark.rdd.RDD[org.apache.spark.sql.Row]) to a Dataframe org.apache.spark.sql.DataFrame. I converted a dataframe to rdd using .rdd. After processing it I want it back in …

Read More