【Weblogic集群】Weblogic Cluster BEA-000116故障处理经验

原文来自：http://www.day-up.com/index.php?ac=article&at=read&did=305

部署环境：Weblogic 10 MP2 for UNIX
实验：
Weblogic Cluster: (MServer1,MServer2) 7222
Multicast IP: 239.192.0.0
AdminServer: 192.168.1.33:7001
MServer1: 192.168.1.33:7201
MServer2: 192.168.1.33:7202
Machine1: (MServer1) NodeManager(192.168.1.33:5556)
Machine2: (MServer2) NodeManager(192.168.1.33:5557)
注意：本实验环境各个Server SSL设置---->Hostname Verification: NONE

描述：
（NodeManager 成功启动）
当启动某ManagedServer时分别报以下错误：
/home/weblogic/bea/user_projects/domains/clustersys/servers/MServer_1/logs/MServer_1.out
Dec 16, 2010 9:12:56 PM CST> <Info> <WebLogicServer> <BEA-000215>
<Loaded License : /home/weblogic/bea/license.bea>
<Dec 16, 2010 9:12:57 PM CST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
<Dec 16, 2010 9:12:57 PM CST> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool>
<Dec 16, 2010 9:12:58 PM CST> <Notice> <Log Management> <BEA-170019> <The server log file
<Dec 16, 2010 9:13:15 PM CST> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>
<Dec 16, 2010 9:13:31 PM CST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STANDBY>
<Dec 16, 2010 9:13:31 PM CST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>

<Dec 16, 2010 9:13:34 PM CST> <Error> <Cluster> <BEA-000116>
<Failed to join cluster Cluster_1 at address 239.192.0.0 due to: java.net.SocketException:
No such device.java.net.SocketException:
No such device
at java.net.PlainDatagramSocketImpl.join(Native Method)
at java.net.PlainDatagramSocketImpl.join(PlainDatagramSocketImpl.java:172)
at java.net.MulticastSocket.joinGroup(MulticastSocket.java:276)
at weblogic.cluster.MulticastFragmentSocket.initializeMulticastSocket(MulticastFragmentSocket.java:112)
at weblogic.cluster.MulticastFragmentSocket.start(MulticastFragmentSocket.java:134)
Truncated. see log file for complete stacktrace
>
<Dec 16, 2010 9:13:34 PM CST> <Critical> <WebLogicServer> <BEA-000362>
<Server failed. Reason: There are 1 nested errors:
java.net.SocketException: No such device
at java.net.PlainDatagramSocketImpl.join(Native Method)
at java.net.PlainDatagramSocketImpl.join(PlainDatagramSocketImpl.java:172)
at java.net.MulticastSocket.joinGroup(MulticastSocket.java:276)
at weblogic.cluster.MulticastFragmentSocket.initializeMulticastSocket(MulticastFragmentSocket.java:112)
at weblogic.cluster.MulticastFragmentSocket.start(MulticastFragmentSocket.java:134)
at weblogic.cluster.FragmentSocketWrapper.start(FragmentSocketWrapper.java:87)
at weblogic.cluster.MulticastManager$1.run(MulticastManager.java:186)
at weblogic.cluster.MulticastManager.startListening(MulticastManager.java:192)
at weblogic.cluster.InboundService.startListening(InboundService.java:40)
at weblogic.cluster.InboundService.start(InboundService.java:29)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:464)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:200)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:172)
>
<Dec 16, 2010 9:13:34 PM CST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>
<Dec 16, 2010 9:13:34 PM CST> <Error> <WebLogicServer> <BEA-000383>
<A critical service failed. The server will shut itself down>
<Dec 16, 2010 9:13:34 PM CST> <Notice> <WebLogicServer> <BEA-000365>
<Server state changed to FORCE_SHUTTING_DOWN>
<Dec 16, 2010 9:13:37 PM> <Debug> <NodeManager> <Waiting for the process to die: 26869>
<Dec 16, 2010 9:13:38 PM> <Info> <NodeManager> <Server failed during startup so will not be restarted>
<Dec 16, 2010 9:13:38 PM> <Debug> <NodeManager>
<runMonitor returned, setting finished=true and notifying waiters>

AdminServer 前台报以下信息：
<Dec 16, 2010 9:13:40 PM CST> <Error> <NodeManager> <BEA-300048> <Unable to start the server MServer_1 :
Exception while starting server 'MServer_1': java.io.IOException: Server failed to start up.
See server output log for more details.>

分析:
依据以下提示，可判断组播地址问题。
Dec 16, 2010 9:13:34 PM CST> <Error> <Cluster> <BEA-000116> <Failed to join cluster Cluster_1
at address 239.192.0.0 due to: java.net.SocketException: No such device.

查看系统路由表:
[root@localhost ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
缺一条组播239.192.0.0路由

手动添加路由:
[root@localhost ~]# route add -host 239.192.0.0 dev eth0
[root@localhost ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
239.192.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0

再启动受控服务器，OK

同样的原因也会导致以下异常抛出： 
注意：
1. 本实验环境各个Server SSL设置---->Hostname Verification:: BEA HOSTNAME Verifier
2. Weblogic 11gr3(10.3) 不适合此环境
 
基于SSL环境下 ，成功启动节点管理器后，受控服务器无法正常启动
 
现象：
当启动MServer_1时，服务器状态：
MServer_1   Machine_1   FAILED_NOT_RESTARTABLE    FAILED

NodeManager 报错：
Dec 16, 2010 11:23:54 PM> <INFO> <clustersys> <States = {MServer_2=UNKNOWN, AdminServer=UNKNOWN, MServer_1=UNKNOWN, domain_bak=UNKNOWN}>
<Dec 16, 2010 11:23:54 PM> <INFO> <clustersys> <States = {MServer_2=UNKNOWN, AdminServer=UNKNOWN, MServer_1=UNKNOWN, domain_bak=UNKNOWN}>
<Dec 16, 2010 11:23:57 PM> <INFO> <clustersys> <States = {MServer_2=UNKNOWN, AdminServer=UNKNOWN,

<Dec 16, 2010 11:23:57 PM> <Info> <clustersys> <MServer_1>
<Dec 16, 2010 11:23:57 PM> <Info> <clustersys> <MServer_1> <Startup configuration properties saved to "/home/weblogic/bea/user_projects/domains/clustersys/servers/MServer_1/data/nodemanager/startup.properties">
<Dec 16, 2010 11:23:57 PM> <Info> <clustersys> <MServer_1> <Rotated server output log to "/home/weblogic/bea/user_projects/domains/clustersys/servers/MServer_1/logs/MServer_1.out00005">
<Dec 16, 2010 11:23:57 PM> <Info> <clustersys> <MServer_1> <Server error log also redirected to server log>

<Dec 16, 2010 11:26:44 PM> <Info> <clustersys> <MServer_1> <Server failed during startup so will not be restarted>
<Dec 16, 2010 11:26:44 PM> <Warning> <Exception while starting server 'MServer_1': java.io.IOException: Server failed to start up. See server output log for more details.>
java.io.IOException: Server failed to start up. See server output log for more details.
        at weblogic.nodemanager.server.ServerManager.start(ServerManager.java:303)
        at weblogic.nodemanager.server.Handler.handleStart(Handler.java:542)
        at weblogic.nodemanager.server.Handler.handleCommand(Handler.java:119)
        at weblogic.nodemanager.server.Handler.run(Handler.java:66)
        at java.lang.Thread.run(Thread.java:595)
<Dec 16, 2010 11:28:18 PM> <INFO> <clustersys> <States = {MServer_2=UNKNOWN, AdminServer=UNKNOWN, MServer_1=FAILED_NOT_RESTARTABLE, domain_bak=UNKNOWN}>
 

控制台：
<Dec 16, 2010 11:26:44 PM CST> <Error> <NodeManager> <BEA-300048> <Unable to start the server MServer_1 : Exception while starting server 'MServer_1': java.io.IOException: Server failed to start up. See server output log for more details.>

查看日志，分析原因：
/home/weblogic/bea/user_projects/domains/clustersys/servers/MServer_1/logs/MServer_1.out

<Dec 16, 2010 11:26:36 PM CST> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>
<Dec 16, 2010 11:26:42 PM CST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STANDBY>
<Dec 16, 2010 11:26:42 PM CST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>

<Dec 16, 2010 11:26:43 PM CST><Error> <Cluster> <BEA-000116> <Failed to join cluster Cluster_1 at address 239.192.0.0 due to:

java.net.SocketException: No such device
        at java.net.PlainDatagramSocketImpl.join(Native Method)
        at java.net.PlainDatagramSocketImpl.join(PlainDatagramSocketImpl.java:172)
        at java.net.MulticastSocket.joinGroup(MulticastSocket.java:276)
        at weblogic.cluster.MulticastFragmentSocket.initializeMulticastSocket(MulticastFragmentSocket.java:112)
        at weblogic.cluster.MulticastFragmentSocket.start(MulticastFragmentSocket.java:134)
        Truncated. see log file for complete stacktrace
>
<Dec 16, 2010 11:26:43 PM CST> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason: There are 1 nested errors:
 
java.net.SocketException: No such device
        at java.net.PlainDatagramSocketImpl.join(Native Method)
        at java.net.PlainDatagramSocketImpl.join(PlainDatagramSocketImpl.java:172)
        at java.net.MulticastSocket.joinGroup(MulticastSocket.java:276)
        at weblogic.cluster.MulticastFragmentSocket.initializeMulticastSocket(MulticastFragmentSocket.java:112)
        at weblogic.cluster.MulticastFragmentSocket.start(MulticastFragmentSocket.java:134)
        at weblogic.cluster.FragmentSocketWrapper.start(FragmentSocketWrapper.java:87)
        at weblogic.cluster.MulticastManager$1.run(MulticastManager.java:186)
        at weblogic.cluster.MulticastManager.startListening(MulticastManager.java:192)
        at weblogic.cluster.InboundService.startListening(InboundService.java:40)
        at weblogic.cluster.InboundService.start(InboundService.java:29)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:464)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:200)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:172)
>
<Dec 16, 2010 11:26:43 PM CST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>
<Dec 16, 2010 11:26:43 PM CST> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>
<Dec 16, 2010 11:26:43 PM CST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>
<Dec 16, 2010 11:26:44 PM> <Debug> <NodeManager> <Waiting for the process to die: 4428>
<Dec 16, 2010 11:26:44 PM> <Info> <NodeManager> <Server failed during startup so will not be restarted>
<Dec 16, 2010 11:26:44 PM> <Debug> <NodeManager> <runMonitor returned, setting finished=true and notifying waiters>
 
解决方法同上，手动添加组播路由

原文来自：

http://www.day-up.com/index.php?ac=article&at=read&did=305

http://blog.csdn.net/frist_huangsuli/article/details/7731615