11gR2 – having a second OCR file on a separate diskgroup
July 26, 2010 16 Comments
Oracle database versions < 11gR2 RAC, the OUI had a screen where we could specify multiple OCR and or Voting disks. Starting with 11gR2 since the OCR and Voting disks are stored on ASM storage we don’t have options to specify multiple OCR or for that matter multiple voting disks. Well do we need multiple disks for OCR and voting disks. with these files stored on ASM, we could take advantage of the ASM mirroring options. Normal Redundancy (two copies of the file), High Redundancy (three copies of the file) or External Redundancy (where ASM does not manage the redundant copy of these files, but the redundancy is maintained /protected at the storage level by mirroring disks.
If the normal redundancy option or the high redundancy option was selected when the disk group that stores the OCR/voting disks, the automatically the disks are mirrored for either normal or high redundancy.
Now what if we do not want to use ASM mirroring.. but would like the clusterware to maintain two or more copies of OCR files on physically different diskgroups. Oracle supports this option, not while installing the Grid infrastructure but after the installation is complete using the ocrconfig utility. Lets go through one such scenario.
OCR is a critical component of the RAC architecture. From the point that the clusterware starts (on server start/reboot)and to start all applications running on the clusterware which includes database, listener, ASM, database services etc, the OCR is consulted by the clusterware for placement of resources. OCR contains all the rules for high availability of these resources. The clusterware will use these definitions for placement of resources when an server / instance crashes. In 11gR2 the OCR contains additional information such as server pool definitions etc.
The first time the clusterware starts it determines the location of the OCR file by checking the following location
1. Request storage administrators to create a new LUN with the same size as the LUN that currently hosts the OCR and voting disks for the 11gR2 cluster
2. Connect to the ASM instance on one for the database servers and create a new disk group and mount the disk group on all instances in the cluster using the following syntax
ALTER DISKGROUP PRD_GRID2 MOUNT
3. Once the diskgroup have been mounted..we can configure the diskgroup for OCR. This configuration requires root privileges.
Connect to the server as root. and execute ocrconfig command from the GRID_HOME /bin directory.[root@prddb3 bin]# ./ocrconfig -add +PRD_GRID2
Note: if the disk group does not have the required privileges then you can get the following error. I ran into this error while configuring OCR on PRD_GRID2 diskgroup.
Also reported in the GRID_HOME/log/prddb3/client directory is a ocrconfig log file that contains the following errors[root@prddb3 bin]# cat /app/grid/product/11.2.0/log/prddb3/client/ocrconfig_4000.logOracle Database 11g Clusterware Release 22.214.171.124.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.2010-07-10 20:38:18.472: [ OCRCONF]ocrconfig starts...2010-07-10 20:38:23.140: [ OCRCLI]proac_replace_dev:[+PRD_GRID2]: Failed. Retval 2010-07-10 20:38:23.140: [ OCRAPI]procr_replace_dev: failed to replace device (8)2010-07-10 20:38:23.140: [ OCRCONF]The new OCR device [+PRD_GRID2] cannot be opened2010-07-10 20:38:23.140: [ OCRCONF]Exiting [status=failed]...
What does this error mean? by default when you create a diskgroup in 11gR2 RAC from the command prompt using SQL plus, the default compatibility attribute of the diskgroup is set to 10.1. This issue does not occur if you create it through ASMCA.[oracle@prddb3]# sqlplus ‘ /as sysasm’NAME COMPATIBILITY DATABASE_COMPATIBILI VOT------------ -------------------- -------------------- ---PRD_DATA 126.96.36.199.0 188.8.131.52.0 NPRD_FRA 184.108.40.206.0 220.127.116.11.0 NPRD_GRID1 18.104.22.168.0 22.214.171.124.0 NPRD_GRID2 10.1.0.0.0 10.1.0.0.0 N
Note The ASM compatibility and the database (RDBMS) compatibility defaults to 10.1. This needs to be changed to 11.2 in order for the clusterware to recognize that this is a 11gR2 ASM configuration.
4. Change the compatibility of the new diskgroup to 11.2 as follows:
5. The commands will change the compatibility levels for the PRD_GRID2 diskgroup and the new values can be verified using the following query:SQL> COL NAME FORMAT A12SQL> COL COMPATIBILITY FORMAT A20SQL> COL DATABASE_COMPATIBILITY FORMAT A20NAME COMPATIBILITY DATABASE_COMPATIBILI------------ -------------------- --------------------PRD_DATA 126.96.36.199.0 188.8.131.52.0PRD_FRA 184.108.40.206.0 220.127.116.11.0PRD_GRID1 18.104.22.168.0 22.214.171.124.0PRD_GRID2 126.96.36.199.0 188.8.131.52.0
6. Once the data has been verified, attempt to configure the diskgroup for OCR. This configuration requires ‘root’ privileges. Connect to the server as root. and execute ocrconfig command from the $GRID_HOME/bin directory.[root@prddb3 bin]# ./ocrconfig -add +PRD_GRID2
7. To verify if the new OCR file is created and the /etc/oracle/ocr.loc file is updated with the new file information.[root@prddb3 bin]# cat /etc/oracle/ocr.lococrconfig_loc=+PRD_GRID1ocrmirrorconfig_loc=+PRD_GRID2
8. The ocrcheck command reflects two OCR locations[root@prddb3 bin]# ./ocrcheckVersion : 3Total space (kbytes) : 262120Used space (kbytes) : 3972Available space (kbytes) : 258148ID : 1288159793Device/File Name : +PRD_GRID1Device/File Name : +PRD_GRID2Cluster registry integrity check succeededLogical corruption check succeeded
9. Apart from the standard utilities used to very the integrity of the OCR file, the $GRID_HOME/log/prddb3/client directory also contains logs reflecting that the change was successful.[root@prddb3 bin]# cat /app/grid/product/11.2.0/log/prddb3/client/ocrconfig_25560.logOracle Database 11g Clusterware Release 184.108.40.206.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.2010-07-10 21:01:00.652: [ OCRCONF]ocrconfig starts...2010-07-10 21:01:13.593: [ OCRCONF]Exiting [status=success]...
10. The clusterware alert log also has an entry indicating successful addition of the OCR disk./app/grid/product/11.2.0/log/prddb3/alertprddb3.log[crsd(27512)]CRS-1007:The OCR/OCR mirror location was replaced by +PRD_GRID2.
11. Lets check if the physical files can be found on the ASM storage. Set the ORACLE_SID to the ASM instance on the server and using asmcmd the following can be found