sparkling.kryo
defregistrator
macro
(defregistrator name & register-impl)
A macro for creating a custom kryo registrator for application that need to register custom kryo serializers.
This macro must be called from a namespace that is AOT compiled. This is not typically an issue since application jars are packaged as uberjars.
Note that we are extending BaseFlamboRegistrator
and not spark’s KryoRegistrator
.
deserialize
(deserialize b)
We piggy back off of spark’s kryo instance from SparkEnv
for the same reasons we do so in serialize
.
serialize
(serialize obj)
We piggy back off of spark’s kryo instance from SparkEnv
since it already has all of our custom serializers and other things we need to serialize our functions.