In Spark, there are 3 primary ways to specify the options for the SparkConf
used to create the SparkContext
:
spark.driver.memory 4g
spark-shell --driver-memory 4g ...
SparkConf
instance before using it to create the SparkContext
:
sparkConf.set( "spark.driver.memory", "4g" )
However, when using spark-shell
, the SparkContext is already created for you by the time you get a shell prompt, in the variable named sc
. When using spark-shell, how do you use option #3 in the list above to set configuration options, if the SparkContext is already created before you have a chance to execute any Scala statements?
In particular, I am trying to use Kyro serialization and GraphX. The prescribed way to use Kryo with GraphX is to execute the following Scala statement when customizing the SparkConf
instance:
GraphXUtils.registerKryoClasses( sparkConf )
How do I accomplish this when running spark-shell
?
Spark 2.0+
You should be able to use SparkSession.conf.set
method to set some configuration option on runtime but it is mostly limited to SQL configuration.
Spark < 2.0
You can simply stop an existing context and create a new one:
import org.apache.spark.{SparkContext, SparkConf}
sc.stop()
val conf = new SparkConf().set("spark.executor.memory", "4g")
val sc = new SparkContext(conf)
As you can read in the official documentation:
Once a SparkConf object is passed to Spark, it is cloned and can no longer be modified by the user. Spark does not support modifying the configuration at runtime.
So as you can see stopping the context it is the only applicable option once shell has been started.
You can always use configuration files or --conf
argument to spark-shell
to set required parameters which will be used be the default context. In case of Kryo you should take a look at:
spark.kryo.classesToRegister
spark.kryo.registrator