How to add the "--deploy-mode cluster" option to my scala code

hatem dagbouj picture hatem dagbouj · May 11, 2017 · Viewed 7.6k times · Source

209/5000 Hello I want to add the option "--deploy-mode cluster" to my code scala:

  val sparkConf = new SparkConfig ().setMaster ("spark: //192.168.60.80:7077")

Without using the shell (the command. \ Spark-submit)

i whant to usage the " spark.submit.deployMode " in scala

Answer

FaigB picture FaigB · May 12, 2017

with SparkConfig:

//set up the spark configuration and create contexts
val sparkConf = new SparkConf().setAppName("SparkApp").setMaster("spark: //192.168.60.80:7077")

val sc = new SparkContext(sparkConf).set("spark.submit.deployMode", "cluster")

with SparkSession:

val spark = SparkSession
   .builder()
   .appName("SparkApp")
   .master("spark: //192.168.60.80:7077")
   .config("spark.submit.deployMode","cluster")
   .enableHiveSupport()
   .getOrCreate()