【报错】spark序列化报错：Kryo serialization failed: Buffer overflow

本文主要是介绍【报错】spark序列化报错：Kryo serialization failed: Buffer overflow，希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！

spark序列化报错

 org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 61186304. To avoid this, increase spark.kryoserializer.buffer.max value.at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:300)at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:313)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)

 

问题解决

the problem is that kryo does not have enough room in its buffer.

• You should be adjusting spark.kryoserializer.buffer.max in your properties file,
• or use --conf "spark.kryoserializer.buffer.max=128m" in your spark-submit command.

这篇关于【报错】spark序列化报错：Kryo serialization failed: Buffer overflow的文章就介绍到这儿，希望我们推荐的文章对编程师们有所帮助！

---