本文主要是介绍Heartbeat3.x应用全攻略之： 测试Heartbeat的HA功能，希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！
Heartbeat3.x应用全攻略之：测试Heartbeat的HA功能
一、启动Heartbeat
1、启动主节点的Heartbeat
Heartbeat安装完成后，自动在/etc/init.d目录下生成了启动脚步文件heartbeat，直接输入/etc/init.d/heartbeat可以看到heartbeat脚本的用法，如下所示：
|
[root@node1 ~]# /etc/init.d/heartbeat Usage: /etc/init.d/heartbeat {start|stop|status|restart|reload|force-reload}
|
因而启动heartbeat可以通过如下命令进行：
|
[root@node1 ~]#service heartbeat start 或者通过 [root@node1 ~]#/etc/init.d/heartbeat start
|
这样就启动了主节点的heartbeat服务
日志信息如下：
|
Feb 5 19:09:48 node1 heartbeat: [22768]: info: glib: ucast: bound send socket to device: eth0 Feb 5 19:09:48 node1 heartbeat: [22768]: info: glib: ucast: bound receive socket to device: eth0 Feb 5 19:09:48 node1 heartbeat: [22768]: info: glib: ucast: started on port 694 interface eth0 to 192.168.12.1 Feb 5 19:09:48 node1 heartbeat: [22768]: info: glib: ping heartbeat started. Feb 5 19:09:48 node1 heartbeat: [22768]: info: glib: ping group heartbeat started. Feb 5 19:09:48 node1 heartbeat: [22768]: info: Local status now set to: 'up' Feb 5 19:09:49 node1 heartbeat: [22768]: info: Link 192.168.12.1:192.168.12.1 up. Feb 5 19:09:49 node1 heartbeat: [22768]: info: Status update for node 192.168.12.1: status ping Feb 5 19:09:49 node1 heartbeat: [22768]: info: Link group1:group1 up. Feb 5 19:09:49 node1 heartbeat: [22768]: info: Status update for node group1: status ping
|
此段日志是Heartbeat在进行初始化配置，例如，heartbeat的心跳时间间隔、UDP广播端口、ping节点的运行状态等，日志信息到这里会暂停，等待120秒之后，heartbeat会继续输出日志，而这个120秒刚好是ha.cf中“initdead”选项的设定时间。此时heartbeat的输出信息如下：
|
Feb 5 19:11:48 node1 heartbeat: [22768]: WARN: node node2: is dead Feb 5 19:11:48 node1 heartbeat: [22768]: info: Comm_now_up(): updating status to active Feb 5 19:11:48 node1 heartbeat: [22768]: info: Local status now set to: 'active' Feb 5 19:11:48 node1 heartbeat: [22768]: info: Starting child client "/usr/local/ha/lib/heartbeat/pingd -m 100 -d 5s" (102,105) Feb 5 19:11:49 node1 heartbeat: [22768]: WARN: No STONITH device configured. Feb 5 19:11:49 node1 heartbeat: [22768]: WARN: Shared disks are not protected. Feb 5 19:11:49 node1 heartbeat: [22768]: info: Resources being acquired from node2. Feb 5 19:11:49 node1 heartbeat: [22794]: info: Starting "/usr/local/ha/lib/heartbeat/pingd -m 100 -d 5s" as uid 102 gid 105 (pid 22794)
|
在上面这段日志中，由于node2还没有启动，所以会给出“node2: is dead”的警告信息，接下来启动了heartbeat插件pingd，由于我们在ha.cf文件中没有配置STONITH，所以日志里也给出了“No STONITH deviceconfigured”的警告提示。
继续看下面的日志：
|
Feb 5 19:11:50 node1 IPaddr[22966]: INFO: Resource is stopped Feb 5 19:11:50 node1 ResourceManager[22938]: info: Running /usr/local/ha/etc/ha.d/resource.d/IPaddr 192.168.12.135 start Feb 5 19:11:50 node1 IPaddr[23029]: INFO: Using calculated nic for 192.168.12.135: eth0 Feb 5 19:11:50 node1 IPaddr[23029]: INFO: Using calculated netmask for 192.168.12.135: 255.255.255.0 Feb 5 19:11:51 node1 pingd: [22794]: info: attrd_lazy_update: Connecting to cluster... 5 retries remaining Feb 5 19:11:51 node1 IPaddr[23029]: INFO: eval ifconfig eth0:0 192.168.12.135 netmask 255.255.255.0 broadcast 192.168.12.255 Feb 5 19:11:51 node1 avahi-daemon[2455]: Registering new address record for 192.168.12.135 on eth0. Feb 5 19:11:51 node1 IPaddr[23015]: INFO: Success Feb 5 19:11:51 node1 Filesystem[23134]: INFO: Resource is stopped Feb 5 19:11:51 node1 ResourceManager[22938]: info: Running /usr/local/ha/etc/ha.d/resource.d/Filesystem /dev/sdf1 /data1 ext3 start Feb 5 19:11:52 node1 Filesystem[23213]: INFO: Running start for /dev/sdf1 on /data1 Feb 5 19:11:52 node1 kernel: kjournald starting. Commit interval 5 seconds Feb 5 19:11:52 node1 kernel: EXT3 FS on sdf1, internal journal Feb 5 19:11:52 node1 kernel: EXT3-fs: mounted filesystem with ordered data mode. Feb 5 19:11:52 node1 Filesystem[23205]: INFO: Success
|
上面这段日志是进行资源的监控和接管，主要完成haresources文件中的设置，在这里是启用集群虚拟IP和挂载磁盘分区此时，通过ifconfig命令查看主节点的网络配置，可以看到，主节点已经自动绑定了集群的IP地址，在HA集群之外的主机上通过ping命令检测集群IP地址192.168.12.135，已经处于可通状态，也就是该地址变的可用。
同时查看磁盘分区的挂载情况，共享磁盘分区/dev/sdf1已经被自动挂载。
|
[root@data-1-1 ha.d]# ip add 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:0c:29:f1:05:57 brd ff:ff:ff:ff:ff:ff inet 192.168.1.55/24 brd 192.168.1.255 scope global eth0 inet 192.168.1.58/24 brd 192.168.1.255 scope global secondary eth0:0 inet 192.168.1.57/24 brd 192.168.1.255 scope global secondary eth0:1 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:0c:29:f1:05:61 brd ff:ff:ff:ff:ff:ff inet 10.1.1.1/24 brd 10.1.1.255 scope global eth1 [root@data-1-1 ha.d]# ip add|grep 192.168.1 inet 192.168.1.55/24 brd 192.168.1.255 scope global eth0 inet 192.168.1.58/24 brd 192.168.1.255 scope global secondary eth0:0 inet 192.168.1.57/24 brd 192.168.1.255 scope global secondary eth0:1 [root@data-1-1 ha.d]#
|
| [root@data-1-1 ha.d]# [root@data-1-1 ha.d]# cat /var/log/ha-log Jun 04 16:37:48 data-1-1 heartbeat: [3870]: info: ************************** Jun 04 16:39:49 data-1-1 heartbeat: [3871]: info: Resources being acquired from data-1-2. harc[3883]: 2015/06/04_16:39:49 info: Running /usr/local/ha/etc/ha.d//rc.d/status status mach_down[3923]: 2015/06/04_16:39:50 info: Taking over resource group IPaddr::192.168.1.58/24/eth0 IPaddr[3940]: 2015/06/04_16:39:50 INFO: Resource is stopped Jun 04 16:39:50 data-1-1 heartbeat: [3884]: info: Local Resource acquisition completed. ResourceManager[3993]: 2015/06/04_16:39:50 info: Acquiring resource group: data-1-2 IPaddr::192.168.1.58/24/eth0 IPaddr[4034]: 2015/06/04_16:39:50 INFO: Resource is stopped ResourceManager[3993]: 2015/06/04_16:39:50 info: Running /usr/local/ha/etc/ha.d/resource.d/IPaddr 192.168.1.58/24/eth0 start IPaddr[4119]: 2015/06/04_16:39:50 INFO: Using calculated netmask for 192.168.1.58: 255.255.255.0 IPaddr[4119]: 2015/06/04_16:39:50 INFO: eval ifconfig eth0:0 192.168.1.58 netmask 255.255.255.0 broadcast 192.168.1.255 IPaddr[4093]: 2015/06/04_16:39:50 INFO: Success mach_down[3923]: 2015/06/04_16:39:50 info: /usr/local/ha/share/heartbeat/mach_down: nice_failback: foreign resources acquired mach_down[3923]: 2015/06/04_16:39:50 info: mach_down takeover complete for node data-1-2. Jun 04 16:39:50 data-1-1 heartbeat: [3871]: info: mach_down takeover complete. Jun 04 16:39:50 data-1-1 heartbeat: [3871]: info: Initial resource acquisition complete (mach_down) harc[4222]: 2015/06/04_16:39:50 info: Running /usr/local/ha/etc/ha.d//rc.d/ip-request-resp ip-request-resp ip-request-resp[4222]: 2015/06/04_16:39:51 received ip-request-resp IPaddr::192.168.1.57/24/eth0 OK yes ResourceManager[4245]: 2015/06/04_16:39:51 info: Acquiring resource group: data-1-1 IPaddr::192.168.1.57/24/eth0 IPaddr[4273]: 2015/06/04_16:39:51 INFO: Resource is stopped ResourceManager[4245]: 2015/06/04_16:39:51 info: Running /usr/local/ha/etc/ha.d/resource.d/IPaddr 192.168.1.57/24/eth0 start IPaddr[4358]: 2015/06/04_16:39:51 INFO: Using calculated netmask for 192.168.1.57: 255.255.255.0 IPaddr[4358]: 2015/06/04_16:39:51 INFO: eval ifconfig eth0:1 192.168.1.57 netmask 255.255.255.0 broadcast 192.168.1.255 IPaddr[4332]: 2015/06/04_16:39:51 INFO: Success Jun 04 16:40:00 data-1-1 heartbeat: [3871]: info: Local Resource acquisition completed. (none) Jun 04 16:40:00 data-1-1 heartbeat: [3871]: info: local resource transition completed. [root@data-1-1 ha.d]# |
3. 列脑
1）节点data-1-1：
| [root@data-1-1 ha.d]# [root@data-1-1 ha.d]# ip add|grep 192.168.1 inet 192.168.1.55/24 brd 192.168.1.255 scope global eth0 inet 192.168.1.58/24 brd 192.168.1.255 scope global secondary eth0:0 inet 192.168.1.57/24 brd 192.168.1.255 scope global secondary eth0:1 [root@data-1-1 ha.d]#
|
2）节点data-1-2：
| [root@data-1-2 ~]# [root@data-1-2 ~]# ip add|grep 192.168.1 inet 192.168.1.56/24 brd 192.168.1.255 scope global eth0 inet 192.168.1.57/24 brd 192.168.1.255 scope global secondary eth0:0 inet 192.168.1.58/24 brd 192.168.1.255 scope global secondary eth0:1 [root@data-1-2 ~]#
|
3）分析处理
通过”pingdata-1-1”和” ping data -1-2”，2节点是可以ping通的；
节点data-1-1和节点data-1-2关闭防火墙；
| [root@data-1-2 ~]# /etc/init.d/iptables status Firewall is stopped. [root@data-1-1 ha.d]# /etc/init.d/iptables stop [root@data-1-1 ha.d]# |
4）heartbeat重启
两节点都关闭；
| [root@data-1-2 ~]# [root@data-1-2 ~]# /etc/init.d/heartbeat stop Stopping High-Availability services: [ OK ] [root@data-1-2 ~]# /etc/init.d/heartbeat start Starting High-Availability services: IPaddr[4368]: INFO: Resource is stopped IPaddr[4414]: INFO: Resource is stopped [ OK ] [root@data-1-2 ~]#
|
2、启动备用节点的Heartbeat
启动备份节点的Heartbeat，与主节点方法一样，使用如下命令：
|
[root@node2 ~]#/etc/init.d/heartbeat start 或者执行 [root@node2 ~]#service heartbeat start
|
这样就启动了备用节点的heartbeat服务，备用节点的heartbeat日志输出信息与主节点相对应，通过“tail -f /var/log/messages”可以看到如下输出：
|
Feb 19 02:52:15 node2 heartbeat: [26880]: info: Pacemaker support: false Feb 19 02:52:15 node2 heartbeat: [26880]: info: ************************** Feb 19 02:52:15 node2 heartbeat: [26880]: info: Configuration validated. Starting heartbeat 3.0.4 Feb 19 02:52:15 node2 heartbeat: [26881]: info: heartbeat: version 3.0.4 Feb 19 02:52:15 node2 heartbeat: [26881]: info: Heartbeat generation: 1297766398 Feb 19 02:52:15 node2 heartbeat: [26881]: info: glib: UDP multicast heartbeat started for group 225.0.0.1 port 694 interface eth0 (ttl=1 loop=0) Feb 19 02:52:15 node2 heartbeat: [26881]: info: glib: ucast: write socket priority set to IPTOS_LOWDELAY on eth0 Feb 19 02:52:15 node2 heartbeat: [26881]: info: glib: ucast: bound send socket to device: eth0 Feb 19 02:52:15 node2 heartbeat: [26881]: info: glib: ping heartbeat started. Feb 19 02:52:15 node2 heartbeat: [26881]: info: glib: ping group heartbeat started. Feb 19 02:52:15 node2 heartbeat: [26881]: info: Local status now set to: 'up' Feb 19 02:52:16 node2 heartbeat: [26881]: info: Link node1:eth0 up. Feb 19 02:52:16 node2 heartbeat: [26881]: info: Status update for node node1: status active Feb 19 02:52:16 node2 heartbeat: [26881]: info: Link 192.168.12.1:192.168.12.1 up. Feb 19 02:52:16 node2 heartbeat: [26881]: info: Status update for node 192.168.12.1: status ping Feb 19 02:52:16 node2 heartbeat: [26881]: info: Link group1:group1 up. Feb 19 02:52:16 node2 harc[26894]: info: Running /usr/local/ha/etc/ha.d//rc.d/status status Feb 19 02:52:17 node2 heartbeat: [26881]: info: Comm_now_up(): updating status to active Feb 19 02:52:17 node2 heartbeat: [26881]: info: Local status now set to: 'active'
|
3 heartbeat实现web服务高可用实战案例1
3.1 安装/配置/启动httpd
1)data-1-1:
| [root@data-1-1 httpd]# cd /var/www/html/ [root@data-1-1 html]# ll total 0 [root@data-1-1 html]# echo 10.1.1.1 > index.html [root@data-1-1 html]# |
2)data-1-2
| [root@data-1-2 httpd]# cd /var/www/html/ [root@data-1-2 html]# echo 10.1.1.2 > index.html [root@data-1-2 html]#
|
3.2 配置hosts
C:\Windows\System32\drivers\etc
| # localhost name resolution is handled within DNS itself. # 127.0.0.1 localhost # ::1 localhost 192.168.1.57 www.pmel.com
|
3.3 测试
1）data-1-1节点heartbeat停止：
| [root@data-1-1 share]# [root@data-1-1 share]# service heartbeat stop Stopping High-Availability services: [ OK ]
|
| [root@data-1-2 html]# ip add|grep 192.168.1 inet 192.168.1.56/24 brd 192.168.1.255 scope global eth0 inet 192.168.1.58/24 brd 192.168.1.255 scope global secondary eth0:0 inet 192.168.1.57/24 brd 192.168.1.255 scope global secondary eth0:1 [root@data-1-2 html]# [root@data-1-2 html]#
|
|
图01 |
2）data-1-1节点heartbeat启动：
| [root@data-1-1 share]# service heartbeat start Starting High-Availability services: IPaddr[13474]: INFO: Resource is stopped IPaddr[13520]: INFO: Resource is stopped [ OK ] [root@data-1-1 share]#
|
| [root@data-1-2 html]# ip add|grep 192.168.1 inet 192.168.1.56/24 brd 192.168.1.255 scope global eth0 inet 192.168.1.58/24 brd 192.168.1.255 scope global secondary eth0:0 [root@data-1-2 html]#
|
|
|
4 heartbeat实现web服务高可用案例2
4.1 停止节点data-1-1和节点data-1-2中的httpd
| [root@data-1-1 share]# [root@data-1-1 share]# service httpd stop Stopping httpd: [ OK ] [root@data-1-1 share]# #开机也不需要自动启动 [root@data-1-1 share]# chkconfig httpd off [root@data-1-1 share]# |
| [root@data-1-2 html]# [root@data-1-2 html]# service httpd stop Stopping httpd: [ OK ] [root@data-1-2 html]# #开机也不需要自动启动 [root@data-1-2 h tml]# chkconfig httpd off [root@data-1-2 html]# |
4.2 备份和修改haresources
1）备份
|
[root@data-1-1 ha.d]# cd /usr/local/ha/etc/ha.d/ [root@data-1-1 ha.d]# cp haresources haresources.01
|
2）修改
|
[root@data-1-1 ha.d]# vi haresources #node-name resource1 resource2 ... resourceN #just.linux-ha.org 135.9.216.110 #just.linux-ha.org 135.9.216.110 http #just.linux-ha.org 135.9.216.110 135.9.215.111 135.9.216.112 httpd #just.linux-ha.org 135.9.216.3/28/eth0/135.9.216.12 httpd #node1 10.0.0.170 Filesystem::/dev/sda1::/data1::ext2 data-1-1 IPaddr::192.168.1.57/24/eth0 httpd data-1-2 IPaddr::192.168.1.58/24/eth0
|
3）查看httpd服务
|
[root@data-1-1 ha.d]# lsof -i :80 [root@data-1-1 ha.d]#
|
4.3 重启启动heartbeat服务
1) 各个节点都执行重启
|
[root@data-1-1 ha.d]# service heartbeat start Starting High-Availability services: IPaddr[14745]: INFO: Resource is stopped IPaddr[14791]: INFO: Resource is stopped [ OK ] [root@data-1-1 ha.d]#
|
2） 查看
节点1:
|
[root@data-1-1 ha.d]# [root@data-1-1 ha.d]# lsof -i :80 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME httpd 15225 root 10u IPv4 24545 0t0 TCP *:http (LISTEN) httpd 15227 apache 10u IPv4 24545 0t0 TCP *:http (LISTEN) httpd 15228 apache 10u IPv4 24545 0t0 TCP *:http (LISTEN) httpd 15229 apache 10u IPv4 24545 0t0 TCP *:http (LISTEN) httpd 15230 apache 10u IPv4 24545 0t0 TCP *:http (LISTEN) httpd 15231 apache 10u IPv4 24545 0t0 TCP *:http (LISTEN) httpd 15232 apache 10u IPv4 24545 0t0 TCP *:http (LISTEN) httpd 15233 apache 10u IPv4 24545 0t0 TCP *:http (LISTEN) httpd 15234 apache 10u IPv4 24545 0t0 TCP *:http (LISTEN) [root@data-1-1 ha.d]#
|
节点2
|
[root@data-1-2 ha.d]# [root@data-1-2 ha.d]# lsof -i :80 [root@data-1-2 ha.d]#
|
4.4 切换服务器测试
1）停止节点data-1-1中的heartbeat服务
[root@data-1-1 ha.d]#
[root@data-1-1 ha.d]# service heartbeatstop
Stopping High-Availability services: [ OK ]
[root@data-1-1 ha.d]# lsof -i :80
[root@data-1-1 ha.d]#
2）查看httpd服务状态
[root@data-1-2ha.d]# lsof -i :80
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
httpd 14912 root 10u IPv4 21811 0t0 TCP *:http (LISTEN)
httpd 14929 apache 10u IPv4 21811 0t0 TCP *:http (LISTEN)
httpd 14930 apache 10u IPv4 21811 0t0 TCP *:http (LISTEN)
httpd 14931 apache 10u IPv4 21811 0t0 TCP *:http (LISTEN)
httpd 14932 apache 10u IPv4 21811 0t0 TCP *:http (LISTEN)
httpd 14933 apache 10u IPv4 21811 0t0 TCP *:http (LISTEN)
httpd 14934 apache 10u IPv4 21811 0t0 TCP *:http (LISTEN)
httpd 14935 apache 10u IPv4 21811 0t0 TCP *:http (LISTEN)
httpd 14936 apache 10u IPv4 21811 0t0 TCP *:http (LISTEN)
[root@data-1-2 ha.d]#
二、测试heartbeat的高可用功能
如何才能得知HA集群是否正常工作，模拟环境测试是个不错的方法，在把Heartbeat高可用性集群放到生产环境中之前，需要做如下几个步骤的测试，从而确定HA是否正常工作：
（1）正常关闭和重启主节点的heartbeat
首先在主节点node1上执行“service heartbeat stop”正常关闭主节点的Heartbeat进程，此时通过ifconfig命令查看主节点网卡信息，正常情况下，应该可以看到主节点已经释放了集群的服务IP地址，同时也释放了挂载的共享磁盘分区，然后查看备份节点，现在备份节点已经接管了集群的服务IP，同时也自动挂载上了共享的磁盘分区。
在这个过程中，使用ping命令对集群服务IP进行测试，可以看到，集群IP一致处于可通状态，并没有任何延时和阻塞现象，也就是说在正常关闭主节点的情况下，主备节点的切换是无缝的，HA对外提供的服务也可以不间断运行。
接着，将主节点heartbeat正常启动，heartbeat启动后，备份节点将自动释放集群服务IP，同时卸载共享磁盘分区，而主节点将再次接管集群服务IP和挂载共享磁盘分区，其实备份节点释放资源与主节点绑定资源是同步进行的。因而，这个过程也是一个无缝切换。
（2）在主节点上拔去网线
拔去主节点连接公共网络的网线后，heartbeat插件ipfail通过ping测试可以立刻检测到网络连接失败，接着自动释放资源，而就在此时，备用节点的ipfail插件也会检测到主节点出现网络故障，在等待主节点释放资源完毕后，备用节点马上接管了集群资源，从而保证了网络服务不间断持续运行。
同理，当主节点网络恢复正常时，由于设置了“auto_failbackon”选项，集群资源将自动从备用节点切会主节点。
（3）关闭主节点的系统
在主节点拔去电源后，备用节点的heartbeat进程会立刻收到主节点已经shutdown的消息，备用节点就开始进行资源的接管，这种情况其实和主节点网络故障的现象类似。
（4）让主节点系统内核崩溃
当主节点系统崩溃后，网络也就失去了响应，那么备用节点的heartbeat进程就会立刻检测到主节点网络故障，然后进行资源切换，但是由于主节点系统内核崩溃，导致自身不能卸载所占有的资源，例如共享磁盘分区、集群服务IP等，那么此时如果没有类似Stonith设备的话，就会出现资源争用的情况，但是如果有Stonith设备，Stonith设备会首先将故障的主节点电源关闭或者重启此节点等操作，这样就让主节点释放了集群资源，当Stonith设备完成所有操作时，备份节点才拿到接管主节点资源的所有权，从而接管主节点的资源。
（完）
三、错误收集
1）
[root@data-1-1 ha.d]# /etc/init.d/heartbeat
Usage: /etc/init.d/heartbeat{start|stop|status|restart|reload|force-reload}
[root@data-1-1 ha.d]#
[root@data-1-1 ha.d]#
[root@data-1-1 ha.d]# service heartbeatstart
Starting High-Availability services:IPaddr[3096]: INFO: Resource is stopped
IPaddr[3142]: INFO: Resource is stopped
[FAILED]
heartbeat[3200]: 2015/06/04_16:09:04 info:Pacemaker support: no
heartbeat[3200]: 2015/06/04_16:09:04 ERROR:Bad permissions on keyfile [/usr/local/ha/etc/ha.d//authkeys], 600 recommended.
heartbeat[3200]: 2015/06/04_16:09:04 ERROR:Authentication configuration error.
heartbeat[3200]: 2015/06/04_16:09:04 ERROR:Configuration error, heartbeat not started.
[root@data-1-1 ha.d]#
处理方法：
| [root@data-1-1 ha.d]# [root@data-1-1 ha.d]# chmod 600 /usr/local/ha/etc/ha.d/authkeys
|
这篇关于Heartbeat3.x应用全攻略之： 测试Heartbeat的HA功能的文章就介绍到这儿，希望我们推荐的文章对编程师们有所帮助！
