Summary
How to start CRS when the private interconnect's netmask has changed. When the subnet has changed, GPnPD will not start. The configuration is stored in the OLR. This HowTo will describe how to update the OLR with the new netmask.
It seems like an unusual scenario, but it has happened to me twice: once was on a production 11.2 database cluster. The infrastructure people dictated a new private interconnect subnet and new IP addresses. The DBA was expected to just fix RAC as this was just a simple infrastructure change, after all!
There are no tools to update the registry settings in the OLR, so we will have to recreate the OLR completely.
Preparation
The netmask has changed from 192.168.0.0/24 to 192.168.99.0/24. CRS will not have started as GPnPD has not started.
[root@h2-o12-cl1 ~]# /u01/app/12.1.0/grid/bin/crsctl status resource ora.gpnpd -init NAME=ora.gpnpd TYPE=ora.gpnp.type TARGET=ONLINE STATE=OFFLINE
The gpnpd.log log file shows the following.
2013-09-08 10:38:13.540: [ GPNP][3569149504]clsgpnpd_lFilterIpTypes: [at clsgpnpd.c:1614] of 3 net interfaces, 3 publics (1 ipv4, 2 ipv6), 0 privates (0 ipv4, 0 ipv6). 2013-09-08 10:38:13.540: [ GPNP][3569149504]clsgpnpd_lCheckIpTypes: [at clsgpnpd.c:1747] (:GPNPD00123:) Fatal: No private GPnP network interfaces found on node. Check network setup.
Create a manual backup of the OLR on each cluster node.
[root@h1-o12-cl1 ~]# /u01/app/12.1.0/grid/bin/ocrconfig -local -manualbackup h1-o12-cl1 2013/09/08 10:53:40 /u01/app/12.1.0/grid/cdata/h1-o12-cl1/backup_20130908_105340.olr h1-o12-cl1 2013/07/16 11:49:13 /u01/app/12.1.0/grid/cdata/h1-o12-cl1/backup_20130716_114913.olr
Shut down CRS on all the cluster nodes.
[root@h1-o12-cl1 ~]# /u01/app/12.1.0/grid/bin/crsctl stop crs -f ... CRS-4133: Oracle High Availability Services has been stopped.
Deconfigure
Run the deconfigure CRS script on all cluster node(s) except the first node.
[root@h2-o12-cl1 ~]# /u01/app/12.1.0/grid/crs/install/rootcrs.pl -deconfig -force -verbose Using configuration parameter file: /u01/app/12.1.0/grid/crs/install/crsconfig_params PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type PRCR-1068 : Failed to query resources Cannot communicate with crsd PRCR-1070 : Failed to check if resource ora.helper is registered Cannot communicate with crsd PRCR-1070 : Failed to check if resource ora.ons is registered Cannot communicate with crsd ACFSK-0039: Module unloaded. ADVMK-0003: Module unloaded. 11:08:29 CLSRSC-336: Successfully deconfigured Oracle clusterware stack on this node
Once the above command finishes on all remote node(s), run the final deconfiguration on the first node.
[root@h1-o12-cl1 ~]# /u01/app/12.1.0/grid/crs/install/rootcrs.pl -deconfig -force -verbose -keepdg -lastnode Using configuration parameter file: /u01/app/12.1.0/grid/crs/install/crsconfig_params 2013/09/08 17:57:06 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf' 2013/09/08 17:57:55 CLSRSC-214: Failed to start 'crsexcl' Failed to start the Clusterware. Last 20 lines of the alert log follow: 2013-09-08 17:57:36.752: [ohasd(3100)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running). ... CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'h1-o12-cl1' has completed CRS-4133: Oracle High Availability Services has been stopped. ACFSK-0039: Module unloaded. ADVMK-0003: Module unloaded. OKSK-00006: Module unloaded. 2013/09/08 17:59:15 CLSRSC-336: Successfully deconfigured Oracle clusterware stack on this node
Move aside any old copies of the GPnP profile. Perform these steps on all cluster nodes.
[root@h1-o12-cl1 gpnp]# cd /u01/app/12.1.0/grid/gpnp [root@h1-o12-cl1 gpnp]# mv -i profiles/peer/profile.xml profiles/peer/profile.xml.0 [root@h1-o12-cl1 gpnp]# mv -i h1-o12-cl1/profiles/peer/profile.xml h1-o12-cl1/profiles/peer/profile.xml.0
Reconfigure
Create a backup of the CRS configuration parameters used during the installation, and make our network subnet change.
[root@h1-o12-cl1 gpnp]# cd /u01/app/12.1.0/grid/crs/install [root@h1-o12-cl1 install]# cp -pi crsconfig_params crsconfig_params.0 [root@h1-o12-cl1 install]# sed -i -s "s/192.168.0/192.168.99/g" crsconfig_params
Run the root script on each cluster node in turn. If the script fails, then run it again until it succeeds.
[root@h1-o12-cl1 install]# time /u01/app/12.1.0/grid/root.sh
Performing root user operation for Oracle 12c
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/12.1.0/grid
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/12.1.0/grid/crs/install/crsconfig_params
2013/09/08 18:06:25 CLSRSC-363: User ignored prerequisites during installation
OLR initialization - successful
root wallet
root wallet cert
root cert export
peer wallet
profile reader wallet
pa wallet
peer wallet keys
pa wallet keys
peer cert request
pa cert request
peer cert
pa cert
peer root cert TP
profile reader root cert TP
pa root cert TP
peer pa cert TP
pa peer cert TP
profile reader pa cert TP
profile reader peer cert TP
peer user cert
pa user cert
2013/09/08 18:07:24 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
OKSK-00028: In memory kernel log buffer address: 0xffff88007fa7ac00, size: 10485760
OKSK-00027: Oracle kernel distributed lock manager hash size is 62500
OKSK-00004: Module load succeeded. Build information: (LOW DEBUG) USM_12.1.0.1.0_LINUX.X64_130110.1 2013/01/13 22:06:03
ADVMK-0001: Module load succeeded. Build information: (LOW DEBUG) - USM_12.1.0.1.0_LINUX.X64_130110.1 built on 2013/01/13 22:16:47.
ACFSK-0037: Module load succeeded. Build information: (LOW DEBUG) USM_12.1.0.1.0_LINUX.X64_130110.1 2013/01/13 22:59:49
OKSK-00010: Persistent OKS log opened at /u01/app/12.1.0/grid/log/h1-o12-cl1/acfs/kernel/acfs.log.0.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'h1-o12-cl1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'h1-o12-cl1' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'h1-o12-cl1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'h1-o12-cl1'
CRS-2676: Start of 'ora.evmd' on 'h1-o12-cl1' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'h1-o12-cl1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'h1-o12-cl1'
CRS-2676: Start of 'ora.gpnpd' on 'h1-o12-cl1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'h1-o12-cl1'
CRS-2672: Attempting to start 'ora.gipcd' on 'h1-o12-cl1'
CRS-2676: Start of 'ora.cssdmonitor' on 'h1-o12-cl1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'h1-o12-cl1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'h1-o12-cl1'
CRS-2672: Attempting to start 'ora.diskmon' on 'h1-o12-cl1'
CRS-2676: Start of 'ora.diskmon' on 'h1-o12-cl1' succeeded
CRS-2676: Start of 'ora.cssd' on 'h1-o12-cl1' succeeded
OKSK-00010: Persistent OKS log opened at /u01/app/12.1.0/grid/log/h1-o12-cl1/acfs/kernel/acfs.log.0.
ASM created and started successfully.
Disk Group C1OCR mounted successfully.
2013/09/08 18:12:28 CLSRSC-428: Existing OCR configuration found, aborting the configuration. Rerun configuration setup after deinstall.
Died at /u01/app/12.1.0/grid/crs/install/crsutils.pm line 3893.
The command '/u01/app/12.1.0/grid/perl/bin/perl -I/u01/app/12.1.0/grid/perl/lib -I/u01/app/12.1.0/grid/crs/install /u01/app/12.1.0/grid/crs/install/rootcrs.pl ' execution failed
real 6m7.023s
user 1m53.731s
sys 1m48.510s
The root script failed first time for this node, so run it again.
[root@h1-o12-cl1 install]# time /u01/app/12.1.0/grid/root.sh
Performing root user operation for Oracle 12c
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/12.1.0/grid
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/12.1.0/grid/crs/install/crsconfig_params
2013/09/08 18:14:17 CLSRSC-363: User ignored prerequisites during installation
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'h1-o12-cl1'
CRS-2673: Attempting to stop 'ora.asm' on 'h1-o12-cl1'
CRS-2673: Attempting to stop 'ora.evmd' on 'h1-o12-cl1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'h1-o12-cl1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'h1-o12-cl1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'h1-o12-cl1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'h1-o12-cl1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'h1-o12-cl1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'h1-o12-cl1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'h1-o12-cl1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'h1-o12-cl1' succeeded
CRS-2677: Stop of 'ora.asm' on 'h1-o12-cl1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'h1-o12-cl1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'h1-o12-cl1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'h1-o12-cl1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'h1-o12-cl1'
CRS-2677: Stop of 'ora.cssd' on 'h1-o12-cl1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'h1-o12-cl1'
CRS-2677: Stop of 'ora.gipcd' on 'h1-o12-cl1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'h1-o12-cl1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'h1-o12-cl1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'h1-o12-cl1' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'h1-o12-cl1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'h1-o12-cl1'
CRS-2676: Start of 'ora.mdnsd' on 'h1-o12-cl1' succeeded
CRS-2676: Start of 'ora.evmd' on 'h1-o12-cl1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'h1-o12-cl1'
CRS-2676: Start of 'ora.gpnpd' on 'h1-o12-cl1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'h1-o12-cl1'
CRS-2672: Attempting to start 'ora.gipcd' on 'h1-o12-cl1'
CRS-2676: Start of 'ora.cssdmonitor' on 'h1-o12-cl1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'h1-o12-cl1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'h1-o12-cl1'
CRS-2672: Attempting to start 'ora.diskmon' on 'h1-o12-cl1'
CRS-2676: Start of 'ora.diskmon' on 'h1-o12-cl1' succeeded
CRS-2676: Start of 'ora.cssd' on 'h1-o12-cl1' succeeded
OKSK-00010: Persistent OKS log opened at /u01/app/12.1.0/grid/log/h1-o12-cl1/acfs/kernel/acfs.log.0.
ASM created and started successfully.
Disk Group C1OCR mounted successfully.
CRS-2672: Attempting to start 'ora.storage' on 'h1-o12-cl1'
CRS-2676: Start of 'ora.storage' on 'h1-o12-cl1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'h1-o12-cl1'
CRS-2676: Start of 'ora.crsd' on 'h1-o12-cl1' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk 4e97f445d0d14f04bfcd356b2d944f9f.
Successful addition of voting disk 2dfd2f8799004f7fbf579dc02cf2b66a.
Successful addition of voting disk f587022f3f934f60bf912fff2b0ed391.
Successfully replaced voting disk group with +C1OCR.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 4e97f445d0d14f04bfcd356b2d944f9f (ORCL:OCR01) [C1OCR]
2. ONLINE 2dfd2f8799004f7fbf579dc02cf2b66a (ORCL:OCR02) [C1OCR]
3. ONLINE f587022f3f934f60bf912fff2b0ed391 (ORCL:OCR03) [C1OCR]
Located 3 voting disk(s).
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'h1-o12-cl1'
CRS-2673: Attempting to stop 'ora.crsd' on 'h1-o12-cl1'
CRS-2677: Stop of 'ora.crsd' on 'h1-o12-cl1' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'h1-o12-cl1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'h1-o12-cl1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'h1-o12-cl1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'h1-o12-cl1'
CRS-2677: Stop of 'ora.storage' on 'h1-o12-cl1' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'h1-o12-cl1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'h1-o12-cl1'
CRS-2673: Attempting to stop 'ora.evmd' on 'h1-o12-cl1'
CRS-2673: Attempting to stop 'ora.asm' on 'h1-o12-cl1'
CRS-2677: Stop of 'ora.mdnsd' on 'h1-o12-cl1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'h1-o12-cl1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'h1-o12-cl1' succeeded
CRS-2677: Stop of 'ora.asm' on 'h1-o12-cl1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'h1-o12-cl1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'h1-o12-cl1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'h1-o12-cl1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'h1-o12-cl1'
CRS-2677: Stop of 'ora.cssd' on 'h1-o12-cl1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'h1-o12-cl1'
CRS-2677: Stop of 'ora.gipcd' on 'h1-o12-cl1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'h1-o12-cl1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
...
CRS-6016: Resource auto-start has completed for server h1-o12-cl1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2013/09/08 18:21:09 CLSRSC-343: Successfully started Oracle clusterware stack
...
Preparing packages for installation...
cvuqdisk-1.0.9-1
PRCN-3004 : Listener LISTENER already exists
PRCC-1015 : LISTENER was already running on h1-o12-cl1
PRCR-1004 : Resource ora.LISTENER.lsnr is already running
2013/09/08 18:22:38 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
real 8m25.651s
user 0m29.763s
sys 0m15.759s
Run the root script on the remaining cluster node(s).
[root@h2-o12-cl1 install]# time /u01/app/12.1.0/grid/root.sh
Performing root user operation for Oracle 12c
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/12.1.0/grid
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/12.1.0/grid/crs/install/crsconfig_params
2013/09/08 18:23:03 CLSRSC-363: User ignored prerequisites during installation
OLR initialization - successful
2013/09/08 18:24:14 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
OKSK-00028: In memory kernel log buffer address: 0xffff880116df1740, size: 10485760
OKSK-00027: Oracle kernel distributed lock manager hash size is 62500
OKSK-00004: Module load succeeded. Build information: (LOW DEBUG) USM_12.1.0.1.0_LINUX.X64_130110.1 2013/01/13 22:06:03
ADVMK-0001: Module load succeeded. Build information: (LOW DEBUG) - USM_12.1.0.1.0_LINUX.X64_130110.1 built on 2013/01/13 22:16:47.
ACFSK-0037: Module load succeeded. Build information: (LOW DEBUG) USM_12.1.0.1.0_LINUX.X64_130110.1 2013/01/13 22:59:49
OKSK-00010: Persistent OKS log opened at /u01/app/12.1.0/grid/log/h2-o12-cl1/acfs/kernel/acfs.log.0.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'h2-o12-cl1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'h2-o12-cl1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'h2-o12-cl1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'h2-o12-cl1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
...
CRS-6016: Resource auto-start has completed for server h2-o12-cl1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2013/09/08 18:32:14 CLSRSC-343: Successfully started Oracle clusterware stack
PRKO-2190 : VIP exists for node h2-o12-cl1, VIP name h2-o12-cl1-vip.example.com
Preparing packages for installation...
cvuqdisk-1.0.9-1
PRCC-1015 : LISTENER was already running on h2-o12-cl1
PRCR-1004 : Resource ora.LISTENER.lsnr is already running
2013/09/08 18:32:48 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
real 9m48.112s
user 1m23.735s
sys 1m48.473s
That's it. All done.
References
Classifications