10gR2., adding dependency to applications/resources managed by CRS

 

Yesterday we had back to back crashes on two our ETST servers.  After analysis we observed that the crash was due to node evictions by CRS due to communication delays with the other servers, reason: faulty network connection.  Outside this, after the crash and  during auto start of the servers, we encountered another issue.  The database would not start automatically.  The CRS attempted to start the database instance however it failed.   After looking at the log files we found that there was an attempt to start the instance while the ASM instance on that server was not completely up.  Not on just on one server but both servers that got evicted.  The instance was then started manually and all was fine.

To fix this issue for the future, after deliberating with the DBA team we decided to add a between the database instance and ASM instance on all the instances. The dependency is to be configured in the OCR file and when CRS starts these resources /applications it will apply the rule to the instance startup process.

This is what was done to implement the dependency..

1.  execute crsstat to check on all the instances configured on the cluster

[oracle@tstdb1 ~]$ crsstat
HA Resource                              Target     State
-----------                              ------     -----
ora.ETST.ETST1.inst ONLINE ONLINE on tstdb1
ora.ETST.ETST2.inst                      ONLINE     ONLINE on tstdb2
ora.ETST.ETST3.inst                      ONLINE     ONLINE on tstdb3
ora.ETST.db                              ONLINE     ONLINE on tstdb1
ora.tstdb1.ASM1.asm ONLINE ONLINE on tstdb1
ora.tstdb1.ETST_TSTDB1.lsnr ONLINE ONLINE on tstdb1
ora.tstdb1.LISTENER_TSTDB1.lsnr ONLINE ONLINE on tstdb1
ora.tstdb1.gsd                           ONLINE     ONLINE on tstdb1
ora.tstdb1.ons                           ONLINE     ONLINE on tstdb1
ora.tstdb1.vip                           ONLINE     ONLINE on tstdb1
ora.tstdb2.ASM2.asm ONLINE ONLINE on tstdb2
ora.tstdb2.ETST_TSTDB2.lsnr ONLINE ONLINE on tstdb2
ora.tstdb2.LISTENER_TSTDB2.lsnr ONLINE ONLINE on tstdb2
ora.tstdb2.gsd                           ONLINE     ONLINE on tstdb2
ora.tstdb2.ons                           ONLINE     ONLINE on tstdb2
ora.tstdb2.vip                           ONLINE     ONLINE on tstdb2
ora.tstdb3.ASM3.asm ONLINE ONLINE on tstdb3
ora.tstdb3.ETST_TSTDB3.lsnr ONLINE ONLINE on tstdb3
ora.tstdb3.LISTENER_TSTDB3.lsnr ONLINE ONLINE on tstdb3
ora.tstdb3.gsd                           ONLINE     ONLINE on tstdb3
ora.tstdb3.ons                           ONLINE     ONLINE on tstdb3
ora.tstdb3.vip                           ONLINE     ONLINE on tstdb3

2.  create a profile file using the current configuration for all instances.  You do not have to connect to the respective servers, this can be done from any server.

$ORA_CRS_HOME/bin/crs_stat -p ora.ETST.ETST1.inst >> /app/oracle/product/crs/crs/public/ora.ETST.ETST1.inst.cap
$ORA_CRS_HOME/bin/crs_stat -p ora.ETST.ETST2.inst >> /app/oracle/product/crs/crs/public/ora.ETST.ETST2.inst.cap
$ORA_CRS_HOME/bin/crs_stat -p ora.ETST.ETST3.inst >> /app/oracle/product/crs/crs/public/ora.ETST.ETST3.inst.cap

Notice the location of the ‘.cap’ file. It is important the cap file is created in that location. Depending on the type of resource there are two locations where the '.cap’ file can be located. The public folder (/app/oracle/product/crs/crs/public/) is used for all resources owned by Oracle and 'private' folder (/app/oracle/product/crs/crs/private/) is used by resources managed by CRS and owned by root.
3.  Update this profile with the dependency

$ORA_CRS_HOME/bin/crs_profile -update ora.ETST.ETST1.inst -r ora.tstdb1.ASM1.asm
$ORA_CRS_HOME/bin/crs_profile -update ora.ETST.ETST2.inst -r ora.tstdb2.ASM2.asm
$ORA_CRS_HOME/bin/crs_profile -update ora.ETST.ETST3.inst -r ora.tstdb3.ASM3.asm

After the above operation change, the profile is updated and the notice that the 'REQUIRED_RESOURCES' parameter is is changed with the ASM instance resource

[oracle@tstdb1 ~]$ cat /app/oracle/product/crs/crs/public/ora.ETST.ETST1.inst.cap
NAME=ora.ETST.ETST1.inst
TYPE=application
ACTION_SCRIPT=/ETST/oracle/product/10.2.0/db/bin/racgwrap
ACTIVE_PLACEMENT=0
AUTO_START=1
CHECK_INTERVAL=600
DESCRIPTION=CRS application for Instance
FAILOVER_DELAY=0
FAILURE_INTERVAL=0
FAILURE_THRESHOLD=0
HOSTING_MEMBERS=tstdb1
OPTIONAL_RESOURCES=
PLACEMENT=restricted
REQUIRED_RESOURCES=ora.tstdb1.ASM1.asm
RESTART_ATTEMPTS=5
SCRIPT_TIMEOUT=600
START_TIMEOUT=0
STOP_TIMEOUT=0
UPTIME_THRESHOLD=7d
USR_ORA_ALERT_NAME=
USR_ORA_CHECK_TIMEOUT=0
USR_ORA_CONNECT_STR=/ as sysdba
USR_ORA_DEBUG=0
USR_ORA_DISCONNECT=false
USR_ORA_FLAGS=
USR_ORA_IF=
USR_ORA_INST_NOT_SHUTDOWN=
USR_ORA_LANG=
USR_ORA_NETMASK=
USR_ORA_OPEN_MODE=
USR_ORA_OPI=false
USR_ORA_PFILE=
USR_ORA_PRECONNECT=none
USR_ORA_SRV=
USR_ORA_START_TIMEOUT=0
USR_ORA_STOP_MODE=immediate
USR_ORA_STOP_TIMEOUT=0
USR_ORA_VIP=

4.  The reminder of the steps should be performed one instance at a time to avoid downtime to the database  All these steps are performed as user 'oracle'

   4(a)..  shutdown instance 1  using server control

  srvctl stop instance -d ETST -i ETST1

   4(b)    unregister the resource/entry for the instance from the OCR

  $ORA_CRS_HOME/bin/crs_unregister ora.ETST.ETST1.inst

   4(c)    register the resource/entry for the instance back to the OCR

  $ORA_CRS_HOME/bin/crs_register ora.ETST.EST1.inst

    4(d)   Start instance 1 using server control 

  srvctl start instance -d ETST -i ETST2

     4(e)   Verify if the change has made it to the OCR file.  The command below will list the content /definitions for the resource.  You can check for  

REQUIRED_RESOURCES=ora.tstdb1.ASM1.asm
  $ORA_CRS_HOME/bin/crs_stat -p ora.ETST.ETST1.inst

      4(f)  repeat steps 4(a) thru 4(e) for other instances in the cluster

About Murali Vallath
Murali Vallath has over 20 years of experience designing and developing databases. He provides independent Oracle consulting services focusing on designing and performance tuning of Oracle databases through Summersky Enterprises (www.summersky.biz). Vallath has successfully completed over 100 successful small, medium and terabyte sized RAC implementations (Oracle 9i, Oracle 10g & Oracle 11gR2 ) for reputed corporate firms. Vallath is a regular speaker at industry conferences and user groups, including the Oracle Open World, UKOUG and IOUG on RAC and Oracle RDBMS performance tuning topics. Vallath's Publications: Author: 1. ‘Oracle Real Application Clusters’ Publisher: Digital Press 2. ‘Oracle 10g RAC, Grid, Services & Clustering’ Publisher: Digital Press. Co-Author 3. 'Automatic Storage Management Publisher: Oracle Press'

2 Responses to 10gR2., adding dependency to applications/resources managed by CRS

  1. Pingback: Blogroll Report 09/07/2010 – 16/07/2010 « Coskan’s Approach to Oracle

  2. odenysenko says:

    Hi.

    It’s really interesting post for adding dependencies between CRS resources,
    but WHY You don’t use provided tools ?

    Just a ONE line to add dependency between DB instance and ASM instance
    srvctl modify instance -d EBSTST -i EBSTST1 -s ASM1

    BTW
    it seems that this database/instances were manually added to OCR
    because DBCA adds dependency automatically

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 189 other followers

%d bloggers like this: