How to prevent java.lang.OutOfMemoryError: PermGen space at Scala compilation?

scala apache-spark memory-management sbt scalatra-sbt
BumbleGee picture BumbleGee · Nov 30, 2011 · Viewed 21.7k times · Source

I have noticed a strange behavior of my scala compiler. It occasionally throws an OutOfMemoryError when compiling a class. Here's the error message:

[info] Compiling 1 Scala source to /Users/gruetter/Workspaces/scala/helloscala/target/scala-2.9.0/test-classes...
java.lang.OutOfMemoryError: PermGen space
Error during sbt execution: java.lang.OutOfMemoryError: PermGen space

It only happens once in a while and the error is usually not thrown on the subsequent compile run. I use Scala 2.9.0 and compile via SBT.

Does anybody have a clue as to what might be the cause for this error? Thanks in advance for your insights.

Answer

michaelliu picture michaelliu · Oct 11, 2012

I use HomeBrew to install sbt on OS X. It supports a SBT_OPTS argument which can be put in ~/.sbtconfig file with export SBT_OPTS=-XX:MaxPermSize=256M.

Related questions

How can I change column types in Spark SQL's DataFrame?

Suppose I'm doing something like: val df = sqlContext.load("com.databricks.spark.csv", Map("path" -> "cars.csv", "header" -> "true")) df.printSchema() root |-- year: string (nullable = true) |-- make: string (nullable = true) |-- model: string (nullable = …

Read More
Spark - load CSV file as DataFrame?

I would like to read a CSV in spark and convert it as DataFrame and store it in HDFS with df.registerTempTable("table_name") I have tried: scala> val df = sqlContext.load("hdfs:///csv/file/dir/file.csv") Error …

Read More
How to convert rdd object to dataframe in spark

How can I convert an RDD (org.apache.spark.rdd.RDD[org.apache.spark.sql.Row]) to a Dataframe org.apache.spark.sql.DataFrame. I converted a dataframe to rdd using .rdd. After processing it I want it back in …

Read More