Kryo serialization refuses to register class

scala kryo
kbirk picture kbirk · Feb 25, 2014 · Viewed 7k times · Source

I'm trying to use kryo serialization with: 

kryo.setRegistrationRequired(true);

I keep getting the following error saying that a certain class is not registered:

java.lang.IllegalArgumentException: Class is not registered: com.my.package.MyClass[]
Note: To register this class use: kryo.register(com.my.package.MyClass[].class);

However, I do register it:

kryo.register(classOf[MyClass[_]])

When I set Log.TRACE() I get the following output:

00:11 TRACE: [kryo] Register class ID 51: com.my.package.MyClass (com.esotericsoftware.kryo.serializers.FieldSerializer)

Why does it say it is not registered, when the trace logger prints that it has been registered. I cannot find any useful documentation on the matter. Has anyone experienced this before? If it helps i'm running Apache Spark v0.8.1

Answer

kbirk picture kbirk · Mar 3, 2014

I figured out what the issue was. I was mistaking the java array syntax [] for the scala generic syntax []. The exception was being being thrown since I did not register an array of MyClass.

So in java

kryo.register( MyClass[].class );

And scala

kryo.register( classOf[ Array[ MyClass[_] ] ] )

Related questions

Customize SparkContext using sparkConf.set(..) when using spark-shell

In Spark, there are 3 primary ways to specify the options for the SparkConf used to create the SparkContext: As properties in the conf/spark-defaults.conf e.g., the line: spark.driver.memory 4g As args to spark-shell or spark-submit e.…

Read More
When to use Kryo serialization in Spark?

I am already compressing RDDs using conf.set("spark.rdd.compress","true") and persist(MEMORY_AND_DISK_SER). Will using Kryo serialization make the program even more efficient, or is it not useful in this case? I know that Kryo …

Read More
Spark Kryo: Register a custom serializer

I have a class that implements a custom Kryo serializer by implementing the read() and write() methods from com.esotericsoftware.kryo.Serializer (see example below). How can I register this custom serializer with Spark? Here is a pseudo-code example of …

Read More