Using G1GC garbage collector with spark 2.3

apache-spark pyspark garbage-collection g1gc
Clock Slave picture Clock Slave · Jun 14, 2018 · Viewed 7.1k times · Source

I am trying to use the G1GC garbage collector for spark job but I get a

Error: Invalid argument to --conf: -XX:+UseG1GC

I tried using these options but haven't been able to get it working

spark-submit --master spark://192.168.60.20:7077 --conf -XX:+UseG1GC /appdata/bblite-codebase/test.py

and 

spark-submit --master spark://192.168.60.20:7077 -XX:+UseG1GC /appdata/bblite-codebase/test.py

What is the correct way to call a G1GC collector from spark?

Answer

user9941512 picture user9941512 · Jun 14, 2018

JVM options should be passed as spark.executor.extraJavaOptions / spark.driver.extraJavaOptions, ie.

 --conf "spark.executor.extraJavaOptions=-XX:+UseG1GC"

Related questions

How to change dataframe column names in pyspark?

I come from pandas background and am used to reading data from CSV files into a dataframe and then simply changing the column names to something useful using the simple command: df.columns = new_column_name_list However, the same …

Read More
Load CSV file with Spark

I'm new to Spark and I'm trying to read CSV data from a file with Spark. Here's what I am doing : sc.textFile('file.csv') .map(lambda line: (line.split(',')[0], line.split(',')[1])) .collect() I would expect …

Read More
How do I add a new column to a Spark DataFrame (using PySpark)?

I have a Spark DataFrame (using PySpark 1.5.1) and would like to add a new column. I've tried the following without any success: type(randomed_hours) # => list # Create in Python and transform to RDD new_col = pd.DataFrame(randomed_hours, …

Read More