oracle 11.2 RAC 系列-安装新主机，识别老存储-1-识别ASM磁盘
在有些场景下，RAC环境中如果主机出现问题，比如硬件故障等，不能启动，我们需要尽快存储上的启动数据库，恢复业务，那么就需要迁移以前的RAC环境到新的主机环境下，我测试了11.2和12.1的RAC，恢复过程还是很快的，基本上就是安装软件的过程，如果真实场景恢复业务，有两种方法：
1，按照我这里的方法重新安装主机，恢复RAC和数据库
2，如果之前有可用的操作系统的备份（比如NBU备份了OS），那么直接使用NBU还原即可
.
我这里测试的是方法1，重新安装11204的GI（Grid Infrastructure）和ORACLE RDBMS软件，然后识别老存储。
测试环境：单节点RAC, 操作系统是OEL Linux 6.6, 数据库版本是11.2.0.4
首先，因为存储使用的是11204的ASM，测试过程只安装11204的GI（Grid Infrastructure）软件，不用OUI配置GI。
[root@lunar5 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@lunar5 ~]#
执行root.sh：
[root@lunar5 ~]# /u01/app/11.2.0.4/grid/root.sh
Performing root user operation for Oracle 11g
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/11.2.0.4/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0.4/grid/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
Installing Trace File Analyzer
OLR initialization - successful
root wallet
root wallet cert
root cert export
peer wallet
profile reader wallet
pa wallet
peer wallet keys
pa wallet keys
peer cert request
pa cert request
peer cert
pa cert
peer root cert TP
profile reader root cert TP
pa root cert TP
peer pa cert TP
pa peer cert TP
profile reader pa cert TP
profile reader peer cert TP
peer user cert
pa user cert
Adding Clusterware entries to upstart
CRS-2672: Attempting to start 'ora.mdnsd' on 'lunar5'
CRS-2676: Start of 'ora.mdnsd' on 'lunar5' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'lunar5'
CRS-2676: Start of 'ora.gpnpd' on 'lunar5' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'lunar5'
CRS-2672: Attempting to start 'ora.gipcd' on 'lunar5'
CRS-2676: Start of 'ora.cssdmonitor' on 'lunar5' succeeded
CRS-2676: Start of 'ora.gipcd' on 'lunar5' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'lunar5'
CRS-2672: Attempting to start 'ora.diskmon' on 'lunar5'
CRS-2676: Start of 'ora.diskmon' on 'lunar5' succeeded
CRS-2676: Start of 'ora.cssd' on 'lunar5' succeeded
ASM created and started successfully.
Disk Group TEMPDG created successfully.
clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4256: Updating the profile
Successful addition of voting disk 381226346c054f62bf5f219ee16d7e25.
Successfully replaced voting disk group with +TEMPDG.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 381226346c054f62bf5f219ee16d7e25 (/dev/asm-diski) [TEMPDG]
Located 1 voting disk(s).
CRS-2672: Attempting to start 'ora.asm' on 'lunar5'
CRS-2676: Start of 'ora.asm' on 'lunar5' succeeded
CRS-2672: Attempting to start 'ora.TEMPDG.dg' on 'lunar5'
CRS-2676: Start of 'ora.TEMPDG.dg' on 'lunar5' succeeded
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@lunar5 ~]#
相应的checkpoint文件内容：
[root@lunar5 ~]# ll /u01/app/grid/Clusterware/ckptGridHA_lunar5.xml
-rw-r--r-- 1 grid oinstall 4176 Feb 15 19:31 /u01/app/grid/Clusterware/ckptGridHA_lunar5.xml
[root@lunar5 ~]#
[root@lunar5 ~]# tail /u01/app/grid/Clusterware/ckptGridHA_lunar5.xml
[root@lunar5 ~]#
这里看到“DESC=”ROOTCRS_NODECONFIG” STATE=”SUCCESS””表示GI已经配置完成。
图形界面点击ok，继续执行其余配置,配置完成后，再次检查checkpoint文件：
[root@lunar5 ~]# ll /u01/app/grid/Clusterware/ckptGridHA_lunar5.xml
-rw-r--r-- 1 grid oinstall 4176 Feb 15 19:31 /u01/app/grid/Clusterware/ckptGridHA_lunar5.xml
[root@lunar5 ~]# cd /u01/app/grid/Clusterware/
[root@lunar5 Clusterware]# ll
total 8
-rw-r--r-- 1 grid oinstall 4176 Feb 15 19:31 ckptGridHA_lunar5.xml
[root@lunar5 Clusterware]#
这里看到，checkpoint文件的日期没有变化，说明checkpoint文件是执行root.sh的时候才有用的，也就是这个过程是11.2中为了方便客户，增加了root.sh的失败后继续配置二设计的，非常体贴的功能。在12.2中，该功能更加方便，他将会只管的告诉你当前配置的检查点情况，如果有些步骤失败后，oracle会自动清除老的配置，以便可以失败安装后不用重装，而是纠正错误后继续配置，类似“断点续传”那种意思。
.
比如，下面是12.2beta上安装RAC时执行root.sh的过程：
[root@lunar1 ~]# /stage/root.sh
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /stage
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /stage/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/grid/crsdata/lunar1/crsconfig/rootcrs_lunar1_2016-01-24_12-44-34AM.log
2016/01/24 12:44:34 CLSRSC-46: Error: '/stage/crs/install/ParentDirPerm_lunar1.txt' does not exist
2016/01/24 12:44:34 CLSRSC-46: Error: '/stage/crs/install/ParentDirPerm_lunar1.txt' does not exist
2016/01/24 12:44:35 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2016/01/24 12:44:35 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2016/01/24 12:44:35 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2016/01/24 12:44:35 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2016/01/24 12:44:37 CLSRSC-363: User ignored prerequisites during installation
2016/01/24 12:44:37 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2016/01/24 12:44:38 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2016/01/24 12:44:39 CLSRSC-594: Executing installation step 5 of 19: 'SaveParamFile'.
2016/01/24 12:44:40 CLSRSC-594: Executing installation step 6 of 19: 'SetupOSD'.
2016/01/24 12:44:41 CLSRSC-594: Executing installation step 7 of 19: 'CheckCRSConfig'.
2016/01/24 12:44:41 CLSRSC-594: Executing installation step 8 of 19: 'SetupLocalGPNP'.
2016/01/24 12:44:42 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2016/01/24 12:44:43 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2016/01/24 12:45:30 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2016/01/24 12:45:31 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2016/01/24 12:45:46 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2016/01/24 12:46:08 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2016/01/24 12:46:11 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
2016/01/24 12:46:11 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2016/01/24 12:46:14 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'lunar1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'lunar1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'lunar1'
CRS-2677: Stop of 'ora.mdnsd' on 'lunar1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'lunar1' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on 'lunar1'
CRS-2677: Stop of 'ora.evmd' on 'lunar1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'lunar1'
CRS-2677: Stop of 'ora.gipcd' on 'lunar1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'lunar1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2016/01/24 12:46:23 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'lunar1'
CRS-2672: Attempting to start 'ora.evmd' on 'lunar1'
CRS-2676: Start of 'ora.mdnsd' on 'lunar1' succeeded
CRS-2676: Start of 'ora.evmd' on 'lunar1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'lunar1'
CRS-2676: Start of 'ora.gpnpd' on 'lunar1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'lunar1'
CRS-2676: Start of 'ora.gipcd' on 'lunar1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'lunar1'
CRS-2676: Start of 'ora.cssdmonitor' on 'lunar1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'lunar1'
CRS-2672: Attempting to start 'ora.diskmon' on 'lunar1'
CRS-2676: Start of 'ora.diskmon' on 'lunar1' succeeded
CRS-2676: Start of 'ora.cssd' on 'lunar1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'lunar1'
CRS-2672: Attempting to start 'ora.ctssd' on 'lunar1'
CRS-2676: Start of 'ora.ctssd' on 'lunar1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'lunar1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'lunar1'
CRS-2676: Start of 'ora.asm' on 'lunar1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'lunar1'
CRS-2676: Start of 'ora.storage' on 'lunar1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'lunar1'
CRS-2676: Start of 'ora.crf' on 'lunar1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'lunar1'
CRS-2676: Start of 'ora.crsd' on 'lunar1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: lunar1
CRS-6016: Resource auto-start has completed for server lunar1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2016/01/24 12:47:34 CLSRSC-343: Successfully started Oracle Clusterware stack
2016/01/24 12:47:34 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'lunar1'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'lunar1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'lunar1'
CRS-2676: Start of 'ora.asm' on 'lunar1' succeeded
CRS-2672: Attempting to start 'ora.CRSDG.dg' on 'lunar1'
CRS-2676: Start of 'ora.CRSDG.dg' on 'lunar1' succeeded
2016/01/24 12:48:43 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.
2016/01/24 12:49:04 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@lunar1 ~]#
可以看到，这个过程比12.2以前的RAC执行root.sh的提示更加清晰了。
好吧，回到我们的环境中，继续检查老的asm磁盘组
[grid@lunar5 ~]$ kfod disks=all ds=true cluster=true status=true
--------------------------------------------------------------------------------
Disk Size Header Path Disk Group User Group
================================================================================
1: 512 Mb MEMBER /dev/asm-diskb DATADG1 grid asmadmin
2: 512 Mb MEMBER /dev/asm-diskc DATADG1 grid asmadmin
3: 512 Mb MEMBER /dev/asm-diskd DATADG1 grid asmadmin
4: 1024 Mb MEMBER /dev/asm-diske CRSDG grid asmadmin
5: 1024 Mb MEMBER /dev/asm-diskf CRSDG grid asmadmin
6: 1024 Mb MEMBER /dev/asm-diskg CRSDG grid asmadmin
7: 5120 Mb MEMBER /dev/asm-diskh DATADG2 grid asmadmin
8: 3072 Mb MEMBER /dev/asm-diski TEMPDG grid asmadmin
--------------------------------------------------------------------------------
ORACLE_SID ORACLE_HOME HOST_NAME
================================================================================
+ASM1 /u01/app/11.2.0.4/grid lunar5
[grid@lunar5 ~]$
将上述磁盘组添加到ASM启动磁盘组的列表中：
[grid@lunar5 ~]$ ss
SQL*Plus: Release 11.2.0.4.0 Production on Mon Feb 15 21:31:29 2016
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> show parameter asm
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
asm_diskgroups string
asm_diskstring string /dev/asm*
asm_power_limit integer 1
asm_preferred_read_failure_groups string
SQL> alter system set asm_diskgroups='TEMPDG','CRSDG', 'DATADG1', 'DATADG2' scope=both;
System altered.
SQL>
对新添加的磁盘组执行mount和dismount后，这些磁盘组就会自动添加到ocr中：
[root@lunar5 ~]# crsctl status res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRSDG.dg
ONLINE ONLINE lunar5
ora.DATADG1.dg
ONLINE ONLINE lunar5
ora.DATADG2.dg
ONLINE ONLINE lunar5
ora.LISTENER.lsnr
ONLINE ONLINE lunar5
ora.TEMPDG.dg
ONLINE ONLINE lunar5
ora.asm
ONLINE ONLINE lunar5 Started
ora.gsd
OFFLINE OFFLINE lunar5
ora.net1.network
ONLINE ONLINE lunar5
ora.ons
ONLINE ONLINE lunar5
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE lunar5
ora.cvu
1 ONLINE ONLINE lunar5
ora.lunar5.vip
1 ONLINE ONLINE lunar5
ora.oc4j
1 ONLINE ONLINE lunar5
ora.scan1.vip
1 ONLINE ONLINE lunar5
[root@lunar5 ~]#
检查ASM磁盘组中的内容（比如包含哪些数据库），因为ASM是采用OMF的，因此大部分情况下，通过磁盘组的目录结构就可以看出来这个存储中以前包含哪些数据库和其他相关信息：
ASMCMD> lsdg
State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED NORMAL N 512 4096 1048576 3072 2769 1024 872 0 N CRSDG/
MOUNTED NORMAL N 512 4096 1048576 1536 1251 512 369 0 N DATADG1/
MOUNTED EXTERN N 512 4096 1048576 5120 3386 0 3386 0 N DATADG2/
MOUNTED EXTERN N 512 4096 1048576 3072 2719 0 2719 0 Y TEMPDG/
ASMCMD> cd +crsdg
ASMCMD> ls
ASMCMD> cd +datadg1
ASMCMD> ls
ASMCMD> cd +datadg2
ASMCMD> ls
LUNAR/
ASMCMD>
这里看到，该存储中，包含了一个数据库，crsdg中的信息已经被我删除了，因此，没有老的ocr和voting disk文件了
现在，已经确定只有datadg2上面有数据，因此可以重组磁盘组，以便合理利用空间（后续我要用这个vm做很多其他测试）：
ASMCMD> lsdg
State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED NORMAL N 512 4096 4194304 1536 1296 512 392 0 N DATADG1/
MOUNTED EXTERN N 512 4096 1048576 5120 3386 0 3386 0 N DATADG2/
MOUNTED EXTERN N 512 4096 1048576 3072 3016 0 3016 0 N DATADG3/
MOUNTED EXTERN N 512 4096 1048576 3072 2719 0 2719 0 Y TEMPDG/
ASMCMD>
使用kfod查看：
[grid@lunar5 ~]$ kfod disks=all ds=true cluster=true status=true
--------------------------------------------------------------------------------
Disk Size Header Path Disk Group User Group
================================================================================
1: 512 Mb MEMBER /dev/asm-diskb DATADG1 grid asmadmin
2: 512 Mb MEMBER /dev/asm-diskc DATADG1 grid asmadmin
3: 512 Mb MEMBER /dev/asm-diskd DATADG1 grid asmadmin
4: 1024 Mb MEMBER /dev/asm-diske DATADG3 grid asmadmin
5: 1024 Mb MEMBER /dev/asm-diskf DATADG3 grid asmadmin
6: 1024 Mb MEMBER /dev/asm-diskg DATADG3 grid asmadmin
7: 5120 Mb MEMBER /dev/asm-diskh DATADG2 grid asmadmin
8: 3072 Mb MEMBER /dev/asm-diski TEMPDG grid asmadmin
--------------------------------------------------------------------------------
ORACLE_SID ORACLE_HOME HOST_NAME
================================================================================
+ASM1 /u01/app/11.2.0.4/grid lunar5
[grid@lunar5 ~]$
确认当前的crs状态都是正常的：
[root@lunar5 ~]# crsctl status res -t
——————————————————————————–
NAME TARGET STATE SERVER STATE_DETAILS
——————————————————————————–
Local Resources
——————————————————————————–
ora.DATADG1.dg
ONLINE ONLINE lunar5
ora.DATADG2.dg
ONLINE ONLINE lunar5
ora.DATADG3.dg
ONLINE ONLINE lunar5
ora.LISTENER.lsnr
ONLINE ONLINE lunar5
ora.TEMPDG.dg
ONLINE ONLINE lunar5
ora.asm
ONLINE ONLINE lunar5 Started
ora.gsd
OFFLINE OFFLINE lunar5
ora.net1.network
ONLINE ONLINE lunar5
ora.ons
ONLINE ONLINE lunar5
——————————————————————————–
Cluster Resources
——————————————————————————–
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE lunar5
ora.cvu
1 ONLINE ONLINE lunar5
ora.lunar5.vip
1 ONLINE ONLINE lunar5
ora.oc4j
1 ONLINE ONLINE lunar5
ora.scan1.vip
1 ONLINE ONLINE lunar5
[root@lunar5 ~]#
至此老存储的ASM磁盘组已经在新的主机上可以识别了。
