spark-shell启动报错：Yarn application has already ended! It might have been killed or unable to launch...

本文主要是介绍spark-shell启动报错：Yarn application has already ended! It might have been killed or unable to launch...，希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！

前半部分转自：https://www.cnblogs.com/tibit/p/7337045.html （后半原创）

spark-shell --master=yarn --deploy-mode=client

启动日志，错误信息如下

其中“Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME”，只是一个警告，官方的解释如下：

To make Spark runtime jars accessible from YARN side, you can specify spark.yarn.archive or spark.yarn.jars. For details please refer to Spark Properties. If neither spark.yarn.archive nor spark.yarn.jars is specified, Spark will create a zip file with all jars under $SPARK_HOME/jars and upload it to the distributed cache.

大概是说：如果 spark.yarn.jars 和 spark.yarn.archive都没配置，会把$SPAR_HOME/jars下面所有jar打包成zip文件，上传到每个工作分区,所以打包分发是自动完成的，没配置这俩参数没关系。

"Yarn application has already ended! It might have been killed or unable to launch application master"，这个可是一个异常，打开mr管理页面，我的是 http://192.168.128.130/8088 ,

重点在红框处，2.2g的虚拟内存实际值，超过了2.1g的上限。也就是说虚拟内存超限，所以contrainer被干掉了，活都是在容器干的，容器被干掉了，还玩个屁。

解决方案

yarn-site.xml 增加配置：

2个配置2选一即可

 1 <!--以下为解决spark-shell 以yarn client模式运行报错问题而增加的配置,估计spark-summit也会有这个问题。2个配置只用配置一个即可解决问题，当然都配置也没问题-->
 2 <!--虚拟内存设置是否生效,若实际虚拟内存大于设置值 ,spark 以client模式运行可能会报错,"Yarn application has already ended! It might have been killed or unable to l"-->
 3 <property>
 4     <name>yarn.nodemanager.vmem-check-enabled</name>
 5     <value>false</value>
 6     <description>Whether virtual memory limits will be enforced for containers</description>
 7 </property>
 8 <!--配置虚拟内存/物理内存的值，默认为2.1,物理内存默认应该是1g，所以虚拟内存是2.1g-->
 9 <property>
10     <name>yarn.nodemanager.vmem-pmem-ratio</name>
11     <value>4</value>
12     <description>Ratio between virtual memory to physical memory when setting memory limits for containers</description>
13 </property>

修改后，启动hadoop，spark-shell.

---------------------------------------------------下面原创------------------------------------------------------------

我在spark1.6的老集群上面的yarn master安装了spark2.3，local模式启动正常，但是spark2.3 on yarn启动（spark）报错信息同上文；区别在于yarn的报错信息：

显然没有那么直接明了的错误提示，进一步查看以下log：HADOOP_HOME/logs/userlogs/application_1522048616169_0028/container_1522048616169_0028_01_000001/stderr

Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/spark/network/util/ByteUnit : Unsupported major.minor version 52.0
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        at org.apache.spark.deploy.history.config$.<init>(config.scala:44)
        at org.apache.spark.deploy.history.config$.<clinit>(config.scala)
        at org.apache.spark.SparkConf$.<init>(SparkConf.scala:635)
        at org.apache.spark.SparkConf$.<clinit>(SparkConf.scala)
        at org.apache.spark.SparkConf.set(SparkConf.scala:94)
        at org.apache.spark.SparkConf$$anonfun$loadFromSystemProperties$3.apply(SparkConf.scala:76)
        at org.apache.spark.SparkConf$$anonfun$loadFromSystemProperties$3.apply(SparkConf.scala:75)
        at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
        at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:221)
        at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
        at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
        at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
        at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
        at org.apache.spark.SparkConf.loadFromSystemProperties(SparkConf.scala:75)
        at org.apache.spark.SparkConf.<init>(SparkConf.scala:70)
        at org.apache.spark.SparkConf.<init>(SparkConf.scala:57)
        at org.apache.spark.deploy.yarn.ApplicationMaster.<init>(ApplicationMaster.scala:62)
        at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:823)
        at org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:854)

        at org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala)

由此可见，是配置的jdk不支持，由于旧的配置引用jdk7，然而spark2.3需要jdk8；因此修改yarn-env.sh

#export JAVA_HOME=/usr/java/jdk1.7.0_55

export JAVA_HOME=/r2/jwb/java/jdk1.8.0_161

yarn没重启，，，继续还是报一样的错。。。yarn重启后再试：

虽然spark session是有了，但是 ，还是有点问题，因为non-zero exit code 1报错还在。先这样吧o(╯□╰)o

这篇关于spark-shell启动报错：Yarn application has already ended! It might have been killed or unable to launch...的文章就介绍到这儿，希望我们推荐的文章对编程师们有所帮助！