oracle中11.2中手工kill所有的CRS进程而不导致主机重启方法
我们都知道，在RAC环境中，如果kill ocssd.bin进程，会引起主机重启。
但是有时候系统已经异常了了，且CRS不能正常关闭，而主机可能是几年没重启的老系统，没人敢重启，现在怎么办？
我们只能尝试手工kill进程的方式，然后手工修复CRS（注意，在10.2 RAC中，只有3个d.bin进程）。
测试环境：操作系统是OEL 6.6
[root@lunar1 ~]# cat /etc/oracle-release
Oracle Linux Server release 6.6
[root@lunar1 ~]#
[root@lunar1 ~]# uname -a
Linux lunar1 3.8.13-44.1.1.el6uek.x86_64 #2 SMP Wed Sep 10 06:10:25 PDT 2014 x86_64 x86_64 x86_64 GNU/Linux
[root@lunar1 ~]#
这套RAC的CRS版本是11.2.0.4：
[root@lunar1 ~]# crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [11.2.0.4.0]
[root@lunar1 ~]# crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [11.2.0.4.0]
[root@lunar1 ~]# crsctl query crs softwareversion
Oracle Clusterware version on node [lunar1] is [11.2.0.4.0]
[root@lunar1 ~]#
注意，由于12.1普通RAC（非Flex Cluster）的情况根本文一样，处理思路和过程也一样。
查看当前CRS的状态：
[root@lunar1 ~]# crsctl status res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRSDG.dg
ONLINE ONLINE lunar1
ONLINE ONLINE lunar2
ora.DATADG1.dg
ONLINE ONLINE lunar1
ONLINE ONLINE lunar2
ora.DATADG2.dg
ONLINE ONLINE lunar1
ONLINE ONLINE lunar2
ora.LISTENER.lsnr
ONLINE ONLINE lunar1
ONLINE ONLINE lunar2
ora.asm
ONLINE ONLINE lunar1 Started
ONLINE ONLINE lunar2 Started
ora.gsd
OFFLINE OFFLINE lunar1
OFFLINE OFFLINE lunar2
ora.net1.network
ONLINE ONLINE lunar1
ONLINE ONLINE lunar2
ora.ons
ONLINE ONLINE lunar1
ONLINE ONLINE lunar2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE lunar2
ora.cvu
1 ONLINE ONLINE lunar2
ora.lunar.db
1 ONLINE ONLINE lunar1 Open
2 ONLINE OFFLINE STARTING
ora.lunar1.vip
1 ONLINE ONLINE lunar1
ora.lunar2.vip
1 ONLINE ONLINE lunar2
ora.oc4j
1 ONLINE ONLINE lunar1
ora.scan1.vip
1 ONLINE ONLINE lunar2
[root@lunar1 ~]#
查看当前所有的CRS进程：
[root@lunar1 ~]# ps -ef|grep d.bin
root 3860 1 0 19:31 ? 00:00:12 /u01/app/11.2.0.4/grid/bin/ohasd.bin reboot
grid 3972 1 0 19:31 ? 00:00:04 /u01/app/11.2.0.4/grid/bin/oraagent.bin
grid 3983 1 0 19:31 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/mdnsd.bin
grid 3994 1 0 19:31 ? 00:00:02 /u01/app/11.2.0.4/grid/bin/gpnpd.bin
root 4004 1 0 19:31 ? 00:00:15 /u01/app/11.2.0.4/grid/bin/orarootagent.bin
grid 4007 1 0 19:31 ? 00:00:12 /u01/app/11.2.0.4/grid/bin/gipcd.bin
root 4019 1 0 19:31 ? 00:00:05 /u01/app/11.2.0.4/grid/bin/osysmond.bin
root 4032 1 0 19:31 ? 00:00:02 /u01/app/11.2.0.4/grid/bin/cssdmonitor
root 4051 1 0 19:31 ? 00:00:02 /u01/app/11.2.0.4/grid/bin/cssdagent
grid 4063 1 0 19:31 ? 00:00:12 /u01/app/11.2.0.4/grid/bin/ocssd.bin
root 4157 1 0 19:31 ? 00:00:06 /u01/app/11.2.0.4/grid/bin/octssd.bin reboot
grid 4180 1 0 19:31 ? 00:00:06 /u01/app/11.2.0.4/grid/bin/evmd.bin
grid 4343 4180 0 19:32 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/evmlogger.bin -o /u01/app/11.2.0.4/grid/evm/log/evmlogger.info -l /u01/app/11.2.0.4/grid/evm/log/evmlogger.log
root 5385 1 1 19:39 ? 00:00:17 /u01/app/11.2.0.4/grid/bin/crsd.bin reboot
grid 5456 1 0 19:39 ? 00:00:04 /u01/app/11.2.0.4/grid/bin/oraagent.bin
root 5473 1 0 19:39 ? 00:00:07 /u01/app/11.2.0.4/grid/bin/orarootagent.bin
grid 5475 1 0 19:39 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/scriptagent.bin
grid 6535 1 0 19:50 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/tnslsnr LISTENER -inherit
oracle 7132 1 0 20:04 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/oraagent.bin
root 7350 7273 0 20:04 pts/2 00:00:00 grep d.bin
[root@lunar1 ~]#
这么多进程，他们的关系参见：11.2 RAC 的启动过程
好吧，我们开始模拟kill进程。首先kill 掉/u01/app/11.2.0.4/grid/bin/ohasd.bin（会自动重启，参见11.2 RAC 的启动过程）
[root@lunar1 ~]# kill -9 3860
[root@lunar1 ~]# ps -ef|grep d.bin
grid 3983 1 0 19:31 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/mdnsd.bin
grid 3994 1 0 19:31 ? 00:00:03 /u01/app/11.2.0.4/grid/bin/gpnpd.bin
grid 4007 1 0 19:31 ? 00:00:13 /u01/app/11.2.0.4/grid/bin/gipcd.bin
root 4019 1 0 19:31 ? 00:00:05 /u01/app/11.2.0.4/grid/bin/osysmond.bin
root 4032 1 0 19:31 ? 00:00:02 /u01/app/11.2.0.4/grid/bin/cssdmonitor
grid 4063 1 0 19:31 ? 00:00:13 /u01/app/11.2.0.4/grid/bin/ocssd.bin
root 4157 1 0 19:31 ? 00:00:06 /u01/app/11.2.0.4/grid/bin/octssd.bin reboot
grid 4180 1 0 19:31 ? 00:00:07 /u01/app/11.2.0.4/grid/bin/evmd.bin
grid 4343 4180 0 19:32 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/evmlogger.bin -o /u01/app/11.2.0.4/grid/evm/log/evmlogger.info -l /u01/app/11.2.0.4/grid/evm/log/evmlogger.log
root 5385 1 1 19:39 ? 00:00:19 /u01/app/11.2.0.4/grid/bin/crsd.bin reboot
grid 5456 1 0 19:39 ? 00:00:04 /u01/app/11.2.0.4/grid/bin/oraagent.bin
root 5473 1 0 19:39 ? 00:00:07 /u01/app/11.2.0.4/grid/bin/orarootagent.bin
grid 5475 1 0 19:39 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/scriptagent.bin
grid 6535 1 0 19:50 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/tnslsnr LISTENER -inherit
oracle 7132 1 0 20:04 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/oraagent.bin
grid 7490 1 0 20:06 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/tnslsnr LISTENER_SCAN1 -inherit
root 7534 2487 14 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/ohasd.bin restart
grid 7571 1 6 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/oraagent.bin
root 7575 1 8 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/orarootagent.bin
root 7578 1 2 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/cssdagent
root 7588 1 3 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/cssdmonitor
root 7676 7273 0 20:07 pts/2 00:00:00 grep d.bin
[root@lunar1 ~]#
然后，我们kill cssdmonitor：
[root@lunar1 ~]# kill -9 4032
-bash: kill: (4032) - No such process
[root@lunar1 ~]#
这里没有这个集成，表示cssdmonitor进程被重启过了：
（参见11.2 RAC 的启动过程）
[root@lunar1 ~]# ps -ef|grep d.bin
grid 3983 1 0 19:31 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/mdnsd.bin
grid 3994 1 0 19:31 ? 00:00:03 /u01/app/11.2.0.4/grid/bin/gpnpd.bin
grid 4007 1 0 19:31 ? 00:00:13 /u01/app/11.2.0.4/grid/bin/gipcd.bin
root 4019 1 0 19:31 ? 00:00:05 /u01/app/11.2.0.4/grid/bin/osysmond.bin
grid 4063 1 0 19:31 ? 00:00:13 /u01/app/11.2.0.4/grid/bin/ocssd.bin
root 4157 1 0 19:31 ? 00:00:06 /u01/app/11.2.0.4/grid/bin/octssd.bin reboot
grid 4180 1 0 19:31 ? 00:00:07 /u01/app/11.2.0.4/grid/bin/evmd.bin
grid 4343 4180 0 19:32 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/evmlogger.bin -o /u01/app/11.2.0.4/grid/evm/log/evmlogger.info -l /u01/app/11.2.0.4/grid/evm/log/evmlogger.log
root 5385 1 1 19:39 ? 00:00:19 /u01/app/11.2.0.4/grid/bin/crsd.bin reboot
grid 5456 1 0 19:39 ? 00:00:05 /u01/app/11.2.0.4/grid/bin/oraagent.bin
root 5473 1 0 19:39 ? 00:00:07 /u01/app/11.2.0.4/grid/bin/orarootagent.bin
grid 5475 1 0 19:39 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/scriptagent.bin
grid 6535 1 0 19:50 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/tnslsnr LISTENER -inherit
oracle 7132 1 0 20:04 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/oraagent.bin
grid 7490 1 0 20:06 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/tnslsnr LISTENER_SCAN1 -inherit
root 7534 2487 3 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/ohasd.bin restart
grid 7571 1 0 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/oraagent.bin
root 7575 1 1 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/orarootagent.bin
root 7578 1 0 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/cssdagent
root 7588 1 0 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/cssdmonitor
root 7740 7273 0 20:07 pts/2 00:00:00 grep d.bin
[root@lunar1 ~]#
上面进程启动时间在20:04~20:07之间的，都是被/u01/app/11.2.0.4/grid/bin/ohasd.bin进程重启后，自动后台重启的。
现在，我们kill mdnsd gpnpd gipcd osysmond。
这4个进程中，前面3个是CRS启动除了ohasd以外，最早启动的几个进程。
如果kill这些进程，ohasd都会重启的：
[root@lunar1 ~]# kill -9 3983 3994 4007 4019
[root@lunar1 ~]# ps -ef|grep d.bin
grid 4063 1 0 19:31 ? 00:00:13 /u01/app/11.2.0.4/grid/bin/ocssd.bin
grid 6535 1 0 19:50 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/tnslsnr LISTENER -inherit
grid 7490 1 0 20:06 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/tnslsnr LISTENER_SCAN1 -inherit
root 7534 2487 2 20:07 ? 00:00:01 /u01/app/11.2.0.4/grid/bin/ohasd.bin restart
grid 7571 1 0 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/oraagent.bin
root 7575 1 1 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/orarootagent.bin
root 7578 1 0 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/cssdagent
root 7588 1 0 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/cssdmonitor
grid 7756 1 1 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/gpnpd.bin
grid 7758 1 1 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/mdnsd.bin
root 7776 7273 0 20:07 pts/2 00:00:00 grep d.bin
[root@lunar1 ~]#
这里我们看到，刚才kill 的4 进程都没起来，怎么回事？
别急，还没到时间，ohasd需要check后才启动，O(∩_∩)O哈哈~
然后，我们kill 监听：
[root@lunar1 ~]# kill -9 6535 7490
[root@lunar1 ~]# ps -ef|grep d.bin
grid 4063 1 0 19:31 ? 00:00:13 /u01/app/11.2.0.4/grid/bin/ocssd.bin
root 7534 2487 2 20:07 ? 00:00:01 /u01/app/11.2.0.4/grid/bin/ohasd.bin restart
grid 7571 1 0 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/oraagent.bin
root 7575 1 0 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/orarootagent.bin
root 7578 1 0 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/cssdagent
root 7588 1 0 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/cssdmonitor
grid 7756 1 1 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/gpnpd.bin
grid 7758 1 0 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/mdnsd.bin
grid 7783 1 2 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/gipcd.bin
root 7785 1 2 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/osysmond.bin
root 7844 1 1 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/ologgerd -m lunar2 -r -d /u01/app/11.2.0.4/grid/crf/db/lunar1
root 7853 1 0 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/octssd.bin
grid 7873 1 1 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/evmd.bin
root 7874 1 14 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/crsd.bin reboot
grid 7944 7873 0 20:08 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/evmlogger.bin -o /u01/app/11.2.0.4/grid/evm/log/evmlogger.info -l /u01/app/11.2.0.4/grid/evm/log/evmlogger.log
grid 7979 1 9 20:08 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/oraagent.bin
grid 7982 1 3 20:08 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/scriptagent.bin
oracle 7986 1 4 20:08 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/oraagent.bin
root 8001 1 3 20:08 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/orarootagent.bin
grid 8025 7979 0 20:08 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/lsnrctl status LISTENER
grid 8028 7979 0 20:08 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/lsnrctl status LISTENER_SCAN1
root 8083 7273 0 20:08 pts/2 00:00:00 grep d.bin
[root@lunar1 ~]#
好吧，看看，刚才kill的进程都被重启了，11.2的RAC真强悍啊。
现在我们kill /etc/init.d/init.ohasd进程：
[root@lunar1 ~]# ps -ef|grep ohasd
root 2487 1 0 19:20 ? 00:00:00 /bin/sh /etc/init.d/init.ohasd run
root 7534 2487 1 20:07 ? 00:00:01 /u01/app/11.2.0.4/grid/bin/ohasd.bin restart
root 8191 7273 0 20:08 pts/2 00:00:00 grep ohasd
[root@lunar1 ~]# kill -9 2487 7534
[root@lunar1 ~]# ps -ef|grep ohasd
root 8239 1 0 20:08 ? 00:00:00 /bin/sh /etc/init.d/init.ohasd run
root 8257 8239 0 20:08 ? 00:00:00 /bin/sh /etc/init.d/init.ohasd run
root 8258 8257 0 20:08 ? 00:00:00 /bin/sh /etc/init.d/init.ohasd run
root 8267 7273 0 20:08 pts/2 00:00:00 grep ohasd
[root@lunar1 ~]# ps -ef|grep ohasd
root 8239 1 0 20:08 ? 00:00:00 /bin/sh /etc/init.d/init.ohasd run
root 8299 7273 0 20:08 pts/2 00:00:00 grep ohasd
[root@lunar1 ~]#
这里我们看到的就是/etc/init.d/init.ohasd被系统自动重启的过程。这些信息会记录在/var/log/message/中：
[root@lunar1 ~]# tail -f /var/log/messages
Jan 24 19:45:31 lunar1 kernel: e1000 0000:00:03.0 eth0: Reset adapter
Jan 24 20:03:50 lunar1 kernel: e1000: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Jan 24 20:03:52 lunar1 kernel: e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Jan 24 20:07:01 lunar1 clsecho: /etc/init.d/init.ohasd: ohasd is restarting 1/10.
Jan 24 20:07:01 lunar1 logger: exec /u01/app/11.2.0.4/grid/perl/bin/perl -I/u01/app/11.2.0.4/grid/perl/lib /u01/app/11.2.0.4/grid/bin/crswrapexece.pl /u01/app/11.2.0.4/grid/crs/install/s_crsconfig_lunar1_env.txt /u01/app/11.2.0.4/grid/bin/ohasd.bin "restart"
Jan 24 20:08:26 lunar1 init: oracle-ohasd main process (2487) killed by KILL signal
Jan 24 20:08:26 lunar1 init: oracle-ohasd main process ended, respawning
Jan 24 20:13:58 lunar1 init: oracle-ohasd main process (8239) killed by KILL signal
Jan 24 20:13:58 lunar1 init: oracle-ohasd main process ended, respawning
Jan 24 20:14:12 lunar1 root: exec /u01/app/11.2.0.4/grid/perl/bin/perl -I/u01/app/11.2.0.4/grid/perl/lib /u01/app/11.2.0.4/grid/bin/crswrapexece.pl /u01/app/11.2.0.4/grid/crs/install/s_crsconfig_lunar1_env.txt /u01/app/11.2.0.4/grid/bin/ohasd.bin "reboot"
^C
[root@lunar1 ~]#
而且他进程都被自动重启了（注意这是crsd进程还没被重启）：
[root@lunar1 ~]# ps -ef|grep d.bin
grid 4063 1 0 19:31 ? 00:00:14 /u01/app/11.2.0.4/grid/bin/ocssd.bin
root 7578 1 0 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/cssdagent
root 7588 1 0 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/cssdmonitor
grid 7756 1 0 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/gpnpd.bin
grid 7758 1 0 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/mdnsd.bin
grid 7783 1 0 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/gipcd.bin
root 7785 1 1 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/osysmond.bin
root 7844 1 0 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/ologgerd -m lunar2 -r -d /u01/app/11.2.0.4/grid/crf/db/lunar1
root 7853 1 0 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/octssd.bin
grid 7873 1 0 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/evmd.bin
root 7874 1 3 20:07 ? 00:00:01 /u01/app/11.2.0.4/grid/bin/crsd.bin reboot
grid 7944 7873 0 20:08 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/evmlogger.bin -o /u01/app/11.2.0.4/grid/evm/log/evmlogger.info -l /u01/app/11.2.0.4/grid/evm/log/evmlogger.log
grid 7979 1 0 20:08 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/oraagent.bin
grid 7982 1 0 20:08 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/scriptagent.bin
oracle 7986 1 0 20:08 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/oraagent.bin
root 8001 1 0 20:08 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/orarootagent.bin
grid 8119 1 0 20:08 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/tnslsnr LISTENER_SCAN1 -inherit
grid 8120 1 0 20:08 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/tnslsnr LISTENER -inherit
root 8321 8319 1 20:08 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/crsctl.bin check has
root 8325 7273 0 20:08 pts/2 00:00:00 grep d.bin
[root@lunar1 ~]#
现在我们依次kill：evmlogger.bin gpnpd.bin mdnsd.bin gipcd.bin evmd.bin oraagent.bin scriptagent.bin oraagent.bin orarootagent.bin和两个lisnterner
[root@lunar1 ~]# kill -9 7944 7756 7758 7783 7873 7979 7982 7986 8001 8119 8120
[root@lunar1 ~]# ps -ef|grep d.bin
grid 4063 1 0 19:31 ? 00:00:14 /u01/app/11.2.0.4/grid/bin/ocssd.bin
root 7578 1 0 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/cssdagent
root 7588 1 0 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/cssdmonitor
root 7785 1 1 20:07 ? 00:00:01 /u01/app/11.2.0.4/grid/bin/osysmond.bin
root 7844 1 0 20:07 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/ologgerd -m lunar2 -r -d /u01/app/11.2.0.4/grid/crf/db/lunar1
root 8593 8591 0 20:09 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/crsctl.bin check has
root 8597 7273 0 20:09 pts/2 00:00:00 grep d.bin
[root@lunar1 ~]#
然后，kill osysmond.bin ologgerd cssdmonitor cssdagent ：
[root@lunar1 ~]# kill -9 7785 7844 7588 7578
[root@lunar1 ~]#
好吧，现在就剩下一个ocssd.bin了：
[root@lunar1 ~]# ps -ef|grep d.bin
grid 4063 1 0 19:31 ? 00:00:14 /u01/app/11.2.0.4/grid/bin/ocssd.bin
root 8629 7273 0 20:10 pts/2 00:00:00 grep d.bin
[root@lunar1 ~]#
现在我们kill 传说中一旦被kill就会引起主机重启的进程 ocssd.bin ：
[root@lunar1 ~]# kill -9 4063
[root@lunar1 ~]#
好了，我们的系统都还好好的，没有重启，资源也都释放干净了：
[root@lunar1 ~]# ipcs -ma
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
------ Semaphore Arrays --------
key semid owner perms nsems
0x00000000 0 root 600 1
0x00000000 65537 root 600 1
------ Message Queues --------
key msqid owner perms used-bytes messages
[root@lunar1 ~]#
[root@lunar1 ~]#
如果要恢复，很简单，只要直接重启crs就ok了：
[root@lunar1 ~]# ps -ef | grep -v grep|grep -E 'init|d.bin|ocls|evmlogger|UID'
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 19:20 ? 00:00:01 /sbin/init
root 2486 1 0 19:20 ? 00:00:00 /bin/sh /etc/init.d/init.tfa run
root 8924 1 0 20:13 ? 00:00:00 /bin/sh /etc/init.d/init.ohasd run
[root@lunar1 ~]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@lunar1 ~]# ps -ef|grep ohasd
root 8924 1 0 20:13 ? 00:00:00 /bin/sh /etc/init.d/init.ohasd run
root 8968 1 4 20:14 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/ohasd.bin reboot
root 9187 7273 0 20:14 pts/2 00:00:00 grep ohasd
[root@lunar1 ~]#
[root@lunar1 ~]# ps -ef|grep d.bin
root 8968 1 0 20:14 ? 00:00:08 /u01/app/11.2.0.4/grid/bin/ohasd.bin reboot
grid 9090 1 0 20:14 ? 00:00:02 /u01/app/11.2.0.4/grid/bin/oraagent.bin
grid 9101 1 0 20:14 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/mdnsd.bin
grid 9112 1 0 20:14 ? 00:00:02 /u01/app/11.2.0.4/grid/bin/gpnpd.bin
root 9122 1 0 20:14 ? 00:00:09 /u01/app/11.2.0.4/grid/bin/orarootagent.bin
grid 9126 1 0 20:14 ? 00:00:08 /u01/app/11.2.0.4/grid/bin/gipcd.bin
root 9139 1 0 20:14 ? 00:00:12 /u01/app/11.2.0.4/grid/bin/osysmond.bin
root 9150 1 0 20:14 ? 00:00:01 /u01/app/11.2.0.4/grid/bin/cssdmonitor
root 9169 1 0 20:14 ? 00:00:01 /u01/app/11.2.0.4/grid/bin/cssdagent
grid 9180 1 0 20:14 ? 00:00:04 /u01/app/11.2.0.4/grid/bin/ocssd.bin
root 9212 1 1 20:14 ? 00:00:28 /u01/app/11.2.0.4/grid/bin/ologgerd -M -d /u01/app/11.2.0.4/grid/crf/db/lunar1
root 9340 1 0 20:18 ? 00:00:02 /u01/app/11.2.0.4/grid/bin/octssd.bin reboot
grid 9363 1 0 20:18 ? 00:00:03 /u01/app/11.2.0.4/grid/bin/evmd.bin
root 9455 1 0 20:18 ? 00:00:09 /u01/app/11.2.0.4/grid/bin/crsd.bin reboot
grid 9532 9363 0 20:18 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/evmlogger.bin -o /u01/app/11.2.0.4/grid/evm/log/evmlogger.info -l /u01/app/11.2.0.4/grid/evm/log/evmlogger.log
grid 9569 1 0 20:18 ? 00:00:02 /u01/app/11.2.0.4/grid/bin/oraagent.bin
grid 9572 1 0 20:18 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/scriptagent.bin
root 9591 1 0 20:18 ? 00:00:05 /u01/app/11.2.0.4/grid/bin/orarootagent.bin
grid 9682 1 0 20:18 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/tnslsnr LISTENER -inherit
grid 9684 1 0 20:18 ? 00:00:00 /u01/app/11.2.0.4/grid/bin/tnslsnr LISTENER_SCAN1 -inherit
oracle 9774 1 0 20:19 ? 00:00:03 /u01/app/11.2.0.4/grid/bin/oraagent.bin
root 10642 7273 0 20:38 pts/2 00:00:00 grep d.bin
[root@lunar1 ~]#
[root@lunar1 ~]# crsctl status res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRSDG.dg
ONLINE ONLINE lunar1
ora.DATADG1.dg
ONLINE ONLINE lunar1
ora.DATADG2.dg
ONLINE ONLINE lunar1
ora.LISTENER.lsnr
ONLINE ONLINE lunar1
ora.asm
ONLINE ONLINE lunar1 Started
ora.gsd
OFFLINE OFFLINE lunar1
ora.net1.network
ONLINE ONLINE lunar1
ora.ons
ONLINE ONLINE lunar1
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE lunar1
ora.cvu
1 ONLINE ONLINE lunar1
ora.lunar.db
1 ONLINE ONLINE lunar1 Open
2 ONLINE OFFLINE
ora.lunar1.vip
1 ONLINE ONLINE lunar1
ora.lunar2.vip
1 ONLINE INTERMEDIATE lunar1 FAILED OVER
ora.oc4j
1 ONLINE ONLINE lunar1
ora.scan1.vip
1 ONLINE ONLINE lunar1
[root@lunar1 ~]#
这里只显示了节点1，因为节点2我关闭了。
测试证明，只要先kill cssdmonitor 和 cssdagent进程(准确的说是cssagent，从那张CRS启动的经典大图上也可以看到这个关系)，再kill ocssd.bin进程，系统是不会重启的。
另外，12.1普通RAC（非Flex Cluster）的情况根本文一样，处理思路和过程也一样。
