Resolving "Kryo serialization failed: Buffer overflow" Spark exception

apache-spark kryo
Raghavendran Pedapati picture Raghavendran Pedapati · Jun 8, 2016 · Viewed 15.1k times · Source

I am trying to run Spark (Java) code and getting the error

org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 27".

Other posts have suggested setting the buffer to its max value. When I tried this with max buffer value of 512MB I got the error

java.lang.ClassNotFoundException: org.apache.spark.serializer.KryoSerializer.buffer.max', '512'

How can I solve this problem?

Answer

nessa.gp picture nessa.gp · Jun 12, 2018

The property name is correct, spark.kryoserializer.buffer.max, the value should include the unit, so in your case is 512m.

Also, dependending where you are setting up the configuration you might have to write --conf spark.kryoserializer.buffer.max=512m. For instance, with a spark-submit or within the <spark-opts>...</spark-opts> of an Oozie worflow action.

Related questions

Customize SparkContext using sparkConf.set(..) when using spark-shell

In Spark, there are 3 primary ways to specify the options for the SparkConf used to create the SparkContext: As properties in the conf/spark-defaults.conf e.g., the line: spark.driver.memory 4g As args to spark-shell or spark-submit e.…

Read More
How Kryo serializer allocates buffer in Spark

Please help to understand how Kryo serializer allocates memory for its buffer. My Spark app fails on a collect step when it tries to collect about 122Mb of data to a driver from workers. com.esotericsoftware.kryo.KryoException: Buffer overflow. …

Read More
Require kryo serialization in Spark (Scala)

I have kryo serialization turned on with this: conf.set( "spark.serializer", "org.apache.spark.serializer.KryoSerializer" ) I want to ensure that a custom class is serialized using kryo when shuffled between nodes. I can register the class with kryo …

Read More