10gR2., adding dependency to applications/resources managed by CRS

 

Yesterday we had back to back crashes on two our ETST servers.  After analysis we observed that the crash was due to node evictions by CRS due to communication delays with the other servers, reason: faulty network connection.  Outside this, after the crash and  during auto start of the servers, we encountered another issue.  The database would not start automatically.  The CRS attempted to start the database instance however it failed.   After looking at the log files we found that there was an attempt to start the instance while the ASM instance on that server was not completely up.  Not on just on one server but both servers that got evicted.  The instance was then started manually and all was fine.

To fix this issue for the future, after deliberating with the DBA team we decided to add a between the database instance and ASM instance on all the instances. The dependency is to be configured in the OCR file and when CRS starts these resources /applications it will apply the rule to the instance startup process.

This is what was done to implement the dependency..

1.  execute crsstat to check on all the instances configured on the cluster

[oracle@tstdb1 ~]$ crsstat
HA Resource                              Target     State
-----------                              ------     -----
ora.ETST.ETST1.inst ONLINE ONLINE on tstdb1
ora.ETST.ETST2.inst                      ONLINE     ONLINE on tstdb2
ora.ETST.ETST3.inst                      ONLINE     ONLINE on tstdb3
ora.ETST.db                              ONLINE     ONLINE on tstdb1
ora.tstdb1.ASM1.asm ONLINE ONLINE on tstdb1
ora.tstdb1.ETST_TSTDB1.lsnr ONLINE ONLINE on tstdb1
ora.tstdb1.LISTENER_TSTDB1.lsnr ONLINE ONLINE on tstdb1
ora.tstdb1.gsd                           ONLINE     ONLINE on tstdb1
ora.tstdb1.ons                           ONLINE     ONLINE on tstdb1
ora.tstdb1.vip                           ONLINE     ONLINE on tstdb1
ora.tstdb2.ASM2.asm ONLINE ONLINE on tstdb2
ora.tstdb2.ETST_TSTDB2.lsnr ONLINE ONLINE on tstdb2
ora.tstdb2.LISTENER_TSTDB2.lsnr ONLINE ONLINE on tstdb2
ora.tstdb2.gsd                           ONLINE     ONLINE on tstdb2
ora.tstdb2.ons                           ONLINE     ONLINE on tstdb2
ora.tstdb2.vip                           ONLINE     ONLINE on tstdb2
ora.tstdb3.ASM3.asm ONLINE ONLINE on tstdb3
ora.tstdb3.ETST_TSTDB3.lsnr ONLINE ONLINE on tstdb3
ora.tstdb3.LISTENER_TSTDB3.lsnr ONLINE ONLINE on tstdb3
ora.tstdb3.gsd                           ONLINE     ONLINE on tstdb3
ora.tstdb3.ons                           ONLINE     ONLINE on tstdb3
ora.tstdb3.vip                           ONLINE     ONLINE on tstdb3
2.  create a profile file using the current configuration for all instances.  You do not have to connect to the respective servers, this can be done from any server.
$ORA_CRS_HOME/bin/crs_stat -p ora.ETST.ETST1.inst >> /app/oracle/product/crs/crs/public/ora.ETST.ETST1.inst.cap
$ORA_CRS_HOME/bin/crs_stat -p ora.ETST.ETST2.inst >> /app/oracle/product/crs/crs/public/ora.ETST.ETST2.inst.cap
$ORA_CRS_HOME/bin/crs_stat -p ora.ETST.ETST3.inst >> /app/oracle/product/crs/crs/public/ora.ETST.ETST3.inst.cap
Notice the location of the ‘.cap’ file. It is important the cap file is created in that location. Depending on the type of resource there are two locations where the '.cap’ file can be located. The public folder (/app/oracle/product/crs/crs/public/) is used for all resources owned by Oracle and 'private' folder (/app/oracle/product/crs/crs/private/) is used by resources managed by CRS and owned by root.

3.  Update this profile with the dependency
$ORA_CRS_HOME/bin/crs_profile -update ora.ETST.ETST1.inst -r ora.tstdb1.ASM1.asm
$ORA_CRS_HOME/bin/crs_profile -update ora.ETST.ETST2.inst -r ora.tstdb2.ASM2.asm
$ORA_CRS_HOME/bin/crs_profile -update ora.ETST.ETST3.inst -r ora.tstdb3.ASM3.asm
After the above operation change, the profile is updated and the notice that the ‘REQUIRED_RESOURCES’ parameter is is changed with the ASM instance resource
[oracle@tstdb1 ~]$ cat /app/oracle/product/crs/crs/public/ora.ETST.ETST1.inst.cap
NAME=ora.ETST.ETST1.inst
TYPE=application
ACTION_SCRIPT=/ETST/oracle/product/10.2.0/db/bin/racgwrap
ACTIVE_PLACEMENT=0
AUTO_START=1
CHECK_INTERVAL=600
DESCRIPTION=CRS application for Instance
FAILOVER_DELAY=0
FAILURE_INTERVAL=0
FAILURE_THRESHOLD=0
HOSTING_MEMBERS=tstdb1
OPTIONAL_RESOURCES=
PLACEMENT=restricted
REQUIRED_RESOURCES=ora.tstdb1.ASM1.asm
RESTART_ATTEMPTS=5
SCRIPT_TIMEOUT=600
START_TIMEOUT=0
STOP_TIMEOUT=0
UPTIME_THRESHOLD=7d
USR_ORA_ALERT_NAME=
USR_ORA_CHECK_TIMEOUT=0
USR_ORA_CONNECT_STR=/ as sysdba
USR_ORA_DEBUG=0
USR_ORA_DISCONNECT=false
USR_ORA_FLAGS=
USR_ORA_IF=
USR_ORA_INST_NOT_SHUTDOWN=
USR_ORA_LANG=
USR_ORA_NETMASK=
USR_ORA_OPEN_MODE=
USR_ORA_OPI=false
USR_ORA_PFILE=
USR_ORA_PRECONNECT=none
USR_ORA_SRV=
USR_ORA_START_TIMEOUT=0
USR_ORA_STOP_MODE=immediate
USR_ORA_STOP_TIMEOUT=0
USR_ORA_VIP=
4.  The reminder of the steps should be performed one instance at a time to avoid downtime to the database  All these steps are performed as user ‘oracle’
   4(a)..  shutdown instance 1  using server control
  srvctl stop instance -d ETST -i ETST1
   4(b)    unregister the resource/entry for the instance from the OCR
  $ORA_CRS_HOME/bin/crs_unregister ora.ETST.ETST1.inst
   4(c)    register the resource/entry for the instance back to the OCR
  $ORA_CRS_HOME/bin/crs_register ora.ETST.EST1.inst
    4(d)   Start instance 1 using server control 
  srvctl start instance -d ETST -i ETST2
     4(e)   Verify if the change has made it to the OCR file.  The command below will list the content /definitions for the resource.  You can check for  
REQUIRED_RESOURCES=ora.tstdb1.ASM1.asm
  $ORA_CRS_HOME/bin/crs_stat -p ora.ETST.ETST1.inst
      4(f)  repeat steps 4(a) thru 4(e) for other instances in the cluster