Sometimes we came into a situation where we’ve to bring down the database instance due to the maintainance purpose so in that case we’ve to bring down the CRS resources gracefully in the target database instance.
STEP 1: Take pre-check of cluster resources running on target database node
[oracle@oratest2 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State
Server State
details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
ONLINE ONLINE
oratest1 STABLE
ONLINE ONLINE
oratest2 STABLE
ora.chad
ONLINE ONLINE
oratest1 STABLE
ONLINE ONLINE
oratest2 STABLE
ora.net1.network
ONLINE ONLINE
oratest1 STABLE
ONLINE ONLINE
oratest2 STABLE
ora.ons
ONLINE ONLINE
oratest1 STABLE
ONLINE ONLINE
oratest2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1
ONLINE ONLINE oratest1 STABLE
2
ONLINE OFFLINE STABLE
3
ONLINE ONLINE oratest2 STABLE
ora.DATA.dg(ora.asmgroup)
1
ONLINE ONLINE oratest1 STABLE
2
OFFLINE OFFLINE STABLE
3
OFFLINE OFFLINE STABLE
ora.LISTENER_SCAN1.lsnr
1
ONLINE ONLINE oratest2 STABLE
ora.LISTENER_SCAN2.lsnr
1
ONLINE ONLINE oratest1 STABLE
ora.LISTENER_SCAN3.lsnr
1
ONLINE ONLINE oratest1 STABLE
ora.asm(ora.asmgroup)
1
ONLINE ONLINE oratest1 Started,STABLE
2
ONLINE OFFLINE STABLE
3
ONLINE OFFLINE oratest2 STARTING
ora.asmnet1.asmnetwork(ora.asmgroup)
1
ONLINE ONLINE oratest1 STABLE
2
ONLINE OFFLINE STABLE
3
ONLINE ONLINE oratest2 STABLE
ora.cvu
1
ONLINE ONLINE oratest1 STABLE
ora.ociprim.db
1
ONLINE OFFLINE STABLE
2
ONLINE ONLINE oratest1 Open,HOME=/u01/app/o
racle/product/19c/db
_1,STABLE
ora.oratest1.vip
1
ONLINE ONLINE oratest1 STABLE
ora.oratest2.vip
1
ONLINE ONLINE oratest2 STABLE
ora.oraview.db
1
ONLINE ONLINE oratest1 Open,HOME=/u01/app/o
racle/product/19c/db
_1,STABLE
2
ONLINE OFFLINE STABLE
ora.qosmserver
1
ONLINE ONLINE oratest1 STABLE
ora.scan1.vip
1
ONLINE ONLINE oratest2 STABLE
ora.scan2.vip
1
ONLINE ONLINE oratest1 STABLE
ora.scan3.vip
1 ONLINE
ONLINE oratest1 STABLE
STEP 2: Take pre-check of database services in the target node
[oracle@oratest2 ~]$
srvctl status service -d OCIPRIM
Service ociprim_svc is
running on instance(s) OCIPRIM1, OCIPRIM2
STEP 3: Enable database blackout to mitigate unnecessary database’s alerts
STEP 4: Relocate service to the surviving node
NOTE: If service/s are working on preferred and available
mode then relocate the service into the available instance
srvctl
relocate service -s OCIPRIM_SVC -t oratest1 -i oratest2 -d OCIPRIM
[oracle@oratest1 ~]$
srvctl status service -d OCIPRIM
Service ociprim_svc is
running on instance(s) OCIPRIM2
STEP 5: take pre-check of pdb mode
select
name,open_mode,restricted from gv$pdbs order by name where con_id>2;
CON_ID|CON_NAME |OPEN
MODE |RESTRICTED
----------|--------------------------------------|----------|----------
3|PDB1 |READ
WRITE|NO
3|PDB1
|READ WRITE|NO
STEP 6: Login as root and stop cluster in the target server
[oracle@oratest2 ~]$
crsctl check crs
CRS-4638: Oracle High
Availability Services is online
CRS-4537: Cluster Ready
Services is online
CRS-4529: Cluster
Synchronization Services is online
CRS-4533: Event Manager is
online
[root@oratest2 ~]# /u01/app/19c/grid/bin/crsctl stop crs
CRS-2791: Starting
shutdown of Oracle High Availability Services-managed resources on 'oratest2'
CRS-2673: Attempting to
stop 'ora.crsd' on 'oratest2'
CRS-2790: Starting
shutdown of Cluster Ready Services-managed resources on server 'oratest2'
CRS-2673: Attempting to
stop 'ora.ociprim.ociprim_svc.svc' on 'oratest2'
CRS-2673: Attempting to
stop 'ora.chad' on 'oratest2'
CRS-2677: Stop of
'ora.ociprim.ociprim_svc.svc' on 'oratest2' succeeded
CRS-2673: Attempting to
stop 'ora.ociprim.db' on 'oratest2'
CRS-2673: Attempting to
stop 'ora.oraview.db' on 'oratest2'
CRS-2677: Stop of
'ora.ociprim.db' on 'oratest2' succeeded
CRS-2677: Stop of
'ora.oraview.db' on 'oratest2' succeeded
CRS-33673: Attempting to
stop resource group 'ora.asmgroup' on server 'oratest2'
CRS-2673: Attempting to
stop 'ora.DATA.dg' on 'oratest2'
CRS-2673: Attempting to
stop 'ora.LISTENER.lsnr' on 'oratest2'
CRS-2677: Stop of
'ora.LISTENER.lsnr' on 'oratest2' succeeded
CRS-2673: Attempting to
stop 'ora.oratest2.vip' on 'oratest2'
CRS-2677: Stop of
'ora.DATA.dg' on 'oratest2' succeeded
CRS-2673: Attempting to
stop 'ora.asm' on 'oratest2'
CRS-2677: Stop of
'ora.oratest2.vip' on 'oratest2' succeeded
CRS-2677: Stop of
'ora.asm' on 'oratest2' succeeded
CRS-2673: Attempting to
stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'oratest2'
CRS-2677: Stop of 'ora.chad'
on 'oratest2' succeeded
CRS-2677: Stop of
'ora.ASMNET1LSNR_ASM.lsnr' on 'oratest2' succeeded
CRS-2673: Attempting to
stop 'ora.asmnet1.asmnetwork' on 'oratest2'
CRS-2677: Stop of
'ora.asmnet1.asmnetwork' on 'oratest2' succeeded
CRS-33677: Stop of
resource group 'ora.asmgroup' on server 'oratest2' succeeded.
CRS-2672: Attempting to
start 'ora.oratest2.vip' on 'oratest1'
CRS-2676: Start of
'ora.oratest2.vip' on 'oratest1' succeeded
CRS-2673: Attempting to
stop 'ora.net1.network' on 'oratest2'
CRS-2677: Stop of
'ora.net1.network' on 'oratest2' succeeded
CRS-2792: Shutdown of
Cluster Ready Services-managed resources on 'oratest2' has completed
CRS-2677: Stop of
'ora.crsd' on 'oratest2' succeeded
CRS-2673: Attempting to
stop 'ora.asm' on 'oratest2'
CRS-2673: Attempting to
stop 'ora.crf' on 'oratest2'
CRS-2673: Attempting to
stop 'ora.mdnsd' on 'oratest2'
CRS-2677: Stop of
'ora.crf' on 'oratest2' succeeded
CRS-2677: Stop of
'ora.mdnsd' on 'oratest2' succeeded
CRS-2677: Stop of
'ora.asm' on 'oratest2' succeeded
CRS-2673: Attempting to
stop 'ora.cluster_interconnect.haip' on 'oratest2'
CRS-2677: Stop of
'ora.cluster_interconnect.haip' on 'oratest2' succeeded
CRS-2673: Attempting to
stop 'ora.ctssd' on 'oratest2'
CRS-2673: Attempting to
stop 'ora.evmd' on 'oratest2'
CRS-2677: Stop of
'ora.ctssd' on 'oratest2' succeeded
CRS-2677: Stop of
'ora.evmd' on 'oratest2' succeeded
CRS-2673: Attempting to
stop 'ora.cssd' on 'oratest2'
CRS-2677: Stop of
'ora.cssd' on 'oratest2' succeeded
CRS-2673: Attempting to
stop 'ora.gipcd' on 'oratest2'
CRS-2673: Attempting to
stop 'ora.gpnpd' on 'oratest2'
CRS-2677: Stop of
'ora.gpnpd' on 'oratest2' succeeded
CRS-2677: Stop of
'ora.gipcd' on 'oratest2' succeeded
CRS-2793: Shutdown of
Oracle High Availability Services-managed resources on 'oratest2' has completed
CRS-4133: Oracle High
Availability Services has been stopped.
STEP 7: Validate whether VIP resource has been failed over successfully.
[oracle@oratest1 ~]$
crsctl stat resource ora.oratest2.vip
NAME=ora.oratest2.vip
TYPE=ora.cluster_vip_net1.type
TARGET=ONLINE
STATE=INTERMEDIATE
on oratest1
STEP 8: Shutdown the target database server
[root@oratest2 ~]# init 0
STEP 9: Once maintainance mode completed, bring up the server
[root@oratest2 ~]# uptime
18:37:56 up 2 min, 2 users,
load average: 2.68, 1.19, 0.45
STEP 10: Once database server comes online, validate CRS status
[oracle@oratest2 ~]$
crsctl check crs
CRS-4638: Oracle High
Availability Services is online
CRS-4537: Cluster Ready
Services is online
CRS-4529: Cluster
Synchronization Services is online
CRS-4533: Event Manager is
online
STEP 11: Validate database and its pdbs status whether they’ve started in their respected mode or not and also check it should not be in restricted mode.
select
name,open_mode,restricted from gv$pdbs order by name where con_id>2;
CON_ID|CON_NAME |OPEN
MODE |RESTRICTED
----------|--------------------------------------|----------|----------
3|PDB1 |READ
WRITE|NO
3|PDB1
|READ WRITE|NO
STEP 12: Relocate/Start the services as per the configuration
[oracle@oratest2 ~]$
srvctl status service -d OCIPRIM
Service ociprim_svc is
running on instance(s) OCIPRIM1,OCIPRIM2
STEP 13: validate VIP resource has been relocate successfully
into its original state
[oracle@oratest2 ~]$
crsctl stat resource ora.oratest2.vip;crsctl stat resource ora.oratest1.vip
NAME=ora.oratest2.vip
TYPE=ora.cluster_vip_net1.type
TARGET=ONLINE
STATE=ONLINE on oratest2
NAME=ora.oratest1.vip
TYPE=ora.cluster_vip_net1.type
TARGET=ONLINE
STATE=ONLINE on oratest1
NOTE: The state should come as online in their respected
node
STEP 14: Remove blackout from the oem.
Very helpful Pradeep.
ReplyDelete