本文主要是介绍spark-shell启动报错:Yarn application has already ended! It might have been killed or unable to launch...,希望对大家解决编程问题提供一定的参考价值,需要的开发者们随着小编来一起学习吧!
前半部分转自:https://www.cnblogs.com/tibit/p/7337045.html (后半原创)
spark-shell --master=yarn --deploy-mode=client
启动日志,错误信息如下
其中“Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME”,只是一个警告,官方的解释如下:
To make Spark runtime jars accessible from YARN side, you can specify spark.yarn.archive
or spark.yarn.jars
. For details please refer to Spark Properties. If neither spark.yarn.archive
nor spark.yarn.jars
is specified, Spark will create a zip file with all jars under $SPARK_HOME/jars
and upload it to the distributed cache.
大概是说:如果 spark.yarn.jars 和 spark.yarn.archive都没配置,会把$SPAR_HOME/jars下面所有jar打包成zip文件,上传到每个工作分区,所以打包分发是自动完成的,没配置这俩参数没关系。
"Yarn application has already ended! It might have been killed or unable to launch application master",这个可是一个异常,打开mr管理页面,我的是 http://192.168.128.130/8088 ,
重点在红框处,2.2g的虚拟内存实际值,超过了2.1g的上限。也就是说虚拟内存超限,所以contrainer被干掉了,活都是在容器干的,容器被干掉了,还玩个屁。
解决方案
yarn-site.xml 增加配置:
2个配置2选一即可
1 <!--以下为解决spark-shell 以yarn client模式运行报错问题而增加的配置,估计spark-summit也会有这个问题。2个配置只用配置一个即可解决问题,当然都配置也没问题--> 2 <!--虚拟内存设置是否生效,若实际虚拟内存大于设置值 ,spark 以client模式运行可能会报错,"Yarn application has already ended! It might have been killed or unable to l"--> 3 <property> 4 <name>yarn.nodemanager.vmem-check-enabled</name> 5 <value>false</value> 6 <description>Whether virtual memory limits will be enforced for containers</description> 7 </property> 8 <!--配置虚拟内存/物理内存的值,默认为2.1,物理内存默认应该是1g,所以虚拟内存是2.1g--> 9 <property> 10 <name>yarn.nodemanager.vmem-pmem-ratio</name> 11 <value>4</value> 12 <description>Ratio between virtual memory to physical memory when setting memory limits for containers</description> 13 </property>
修改后,启动hadoop,spark-shell.
---------------------------------------------------下面原创------------------------------------------------------------
我在spark1.6的老集群上面的yarn master安装了spark2.3,local模式启动正常,但是spark2.3 on yarn启动(spark)报错信息同上文;区别在于yarn的报错信息:
显然没有那么直接明了的错误提示,进一步查看以下log:HADOOP_HOME/logs/userlogs/application_1522048616169_0028/container_1522048616169_0028_01_000001/stderr
Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/spark/network/util/ByteUnit : Unsupported major.minor version 52.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at org.apache.spark.deploy.history.config$.<init>(config.scala:44)
at org.apache.spark.deploy.history.config$.<clinit>(config.scala)
at org.apache.spark.SparkConf$.<init>(SparkConf.scala:635)
at org.apache.spark.SparkConf$.<clinit>(SparkConf.scala)
at org.apache.spark.SparkConf.set(SparkConf.scala:94)
at org.apache.spark.SparkConf$$anonfun$loadFromSystemProperties$3.apply(SparkConf.scala:76)
at org.apache.spark.SparkConf$$anonfun$loadFromSystemProperties$3.apply(SparkConf.scala:75)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:221)
at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.SparkConf.loadFromSystemProperties(SparkConf.scala:75)
at org.apache.spark.SparkConf.<init>(SparkConf.scala:70)
at org.apache.spark.SparkConf.<init>(SparkConf.scala:57)
at org.apache.spark.deploy.yarn.ApplicationMaster.<init>(ApplicationMaster.scala:62)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:823)
at org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:854)
at org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala)
由此可见,是配置的jdk不支持,由于旧的配置引用jdk7,然而spark2.3需要jdk8;因此修改yarn-env.sh
#export JAVA_HOME=/usr/java/jdk1.7.0_55
export JAVA_HOME=/r2/jwb/java/jdk1.8.0_161
yarn没重启,,,继续还是报一样的错。。。yarn重启后再试:
虽然spark session是有了,但是 ,还是有点问题,因为non-zero exit code 1报错还在。先这样吧o(╯□╰)o
这篇关于spark-shell启动报错:Yarn application has already ended! It might have been killed or unable to launch...的文章就介绍到这儿,希望我们推荐的文章对编程师们有所帮助!