How to pass external parameters through Spark submit

Question

In my Application, i need to connect to the database so i need to pass IP address and database name when application is submitted.

I submit the application as follows: :

./spark-submit --class class name --master spark://localhost:7077 \
--deploy-mode client /home/hadoop/myjar.jar

Answer 1

If you check the official documentation you'll see that spark-submit has following syntax:

./bin/spark-submit \
  --class <main-class>
  --master <master-url> \
  --deploy-mode <deploy-mode> \
  --conf <key>=<value> \
  ... # other options
  <application-jar> \
  [application-arguments]

You can use either application-arguments and conf to pass required configuration to the main method and SparkConf respectively.