Is it possible to get the current spark context settings in PySpark?

apache-spark config pyspark
whisperstream picture whisperstream · May 31, 2015 · Viewed 95.3k times · Source

I'm trying to get the path to spark.worker.dir for the current sparkcontext.

If I explicitly set it as a config param, I can read it back out of SparkConf, but is there anyway to access the complete config (including all defaults) using PySpark?

Answer

Kevad picture Kevad · Jul 11, 2017

Spark 2.1+

spark.sparkContext.getConf().getAll() where spark is your sparksession (gives you a dict with all configured settings)

Related questions

How to change memory per node for apache spark worker

I am configuring an Apache Spark cluster. When I run the cluster with 1 master and 3 slaves, I see this on the master monitor page: Memory 2.0 GB (512.0 MB Used) 2.0 GB (512.0 MB Used) 6.0 GB (512.0 MB Used) I want to increase the used …

Read More
How do I run multiple spark applications in parallel in standalone master

Using Spark(1.6.1) standalone master, I need to run multiple applications on same spark master. All application submitted after first one, keep on holding 'WAIT' state always. I also observed, the one running holds all cores sum of workers. I already …

Read More
How can I change column types in Spark SQL's DataFrame?

Suppose I'm doing something like: val df = sqlContext.load("com.databricks.spark.csv", Map("path" -> "cars.csv", "header" -> "true")) df.printSchema() root |-- year: string (nullable = true) |-- make: string (nullable = true) |-- model: string (nullable = …

Read More