I want to store the Spark arguments such as input file, output file into a Java property files and pass that file into Spark Driver. I'm using spark-submit for submitting the job but couldn't find a parameter to pass the properties file. Have you got any suggestions?
here i found one solution:
props file : (mypropsfile.conf) // note: prefix your key with "spark." else props will be ignored.
spark.myapp.input /input/path
spark.myapp.output /output/path
launch
$SPARK_HOME/bin/spark-submit --properties-file mypropsfile.conf
how to call in code :( inside code)
sc.getConf.get("spark.driver.host") // localhost
sc.getConf.get("spark.myapp.input") // /input/path
sc.getConf.get("spark.myapp.output") // /output/path