1. Introduction
This brief HOWTO is based on the excellent article from Oracle detailing how to build a RAC cluster on GNU/Linux using ASM as the storage medium. It also pulls some information from this other great article though I used an iSCSI SAN for shared storage.
It is assumed that the Etch distribution of Debian GNU/Linux is used. The Linux kernel that this was tested on was version 2.6.18-4-686.
2. Working around assumptions
2.1. Packages
You will need some extra packages that may not have been installed. For the Oracle Universal Installer you will need to install libxp6, libstdc++5, unzip and build-essential. For CRS you will need arping and iproute, and for the database you will need to install libaio1 as well.
2.2. File paths
The installer assumes that awk and basename are installed in /bin rather than /usr/bin. CRS, specifically $ORA_CRS_HOME/bin/racgvip, assumes that sort is also in /bin. Therefore, create soft links, i.e.
ln -s /usr/bin/awk /bin
ln -s /usr/bin/basename /bin
ln -s /usr/bin/sort /bin
If you find that your virtual host IP addresses keep bouncing from node to node then you need to create the soft link for sort as detailed above.
2.3. Init
The guide assumes that the system will be at run level 3, but Debian GNU/Linux runs at run level 2. Change id:2:initdefault: in /etc/inittab to id:3:initdefault: and switch run levels by issuing telinit 3.
2.4. Kernel modules
The raw kernel module is not loaded by default, so add raw to /etc/modules and run modprobe raw to load it immediately.
Udev only creates the /dev/rawctl device node, so a script is required to create the remaining device nodes that act as the raw devices. The following script, named /etc/init.d/rawdevices will suffice for this purpose:
#! /bin/sh
set -e
# /etc/init.d/rawdevices: start and stop the raw(8) devices
test -x /sbin/raw || exit 0
export PATH="${PATH:+$PATH:}/usr/sbin:/sbin"
case "$1" in
start)
echo -n "Starting RAW devices: "
mkdir -p /dev/raw
for i in 1 2 3 4 5 6 7 8 9 10; do
mknod /dev/raw/raw$i c 162 $i
chown root:oinstall /dev/raw/raw$i
chmod 660 /dev/raw/raw$i
done
# OCR devices
raw /dev/raw/raw1 /dev/sda1
raw /dev/raw/raw2 /dev/sdb1
# Voting disk devices
raw /dev/raw/raw3 /dev/sdc1
raw /dev/raw/raw4 /dev/sdd1
raw /dev/raw/raw5 /dev/sde1
# ASM disk devices
raw /dev/raw/raw6 /dev/sda2
raw /dev/raw/raw7 /dev/sdb2
raw /dev/raw/raw8 /dev/sdc2
raw /dev/raw/raw9 /dev/sdd2
raw /dev/raw/raw10 /dev/sde2
echo "rawdevices."
;;
stop)
echo -n "Stopping RAW devices: "
for i in 1 2 3 4 5 6 7 8 9 10; do
raw /dev/raw/raw$i 0 0
rm -f /dev/raw/raw$i
done
rmdir /dev/raw
echo "rawdevices."
;;
*)
echo "Usage: /etc/init.d/rawdevices {start|stop}"
exit 1
esac
exit 0
Install the script with update-rc.d rawdevices defaults and start it with /etc/init.d/rawdevices start. It is not sufficient to call the standard MAKEDEV raw script as that script checks if Udev is running and refuses to create the device nodes in /dev if it is.
2.5. Root scripts
The Oracle root.sh scripts must be all altered before running. Any references to chgrp nobody must be changed to chgrp nogroup.
3. Installation
3.1. CRS
3.1.1. Startup
The cluster scripts are not set to start on boot properly. Run update-rc.d init.crs start 96 3 4 5 . stop 96 0 1 2 6 . to resolve this. This will also allow you to shut the cluster down on demand by changing to run level 2, and restart it by changing back to run level 3.
3.1.2. Virtual node IP addresses
There may be issues installing CRS because the virtual node IP addresses are in the private subnet ranges. To override this, run $ORA_CRS_HOME/bin/vipca as the root user on any affected nodes.
3.1.3. Virtual node IP network interfaces
Oracle CRS virtual node IP addresses use IPv4, and so CRS must create a network interface alias to assign more than one IP address to a network interface. If you use eth0 for your public interface, you would expect to see interface eth0:0 holding the virtual IP. Unfortunately, you will see many more interface aliases than this. This is due to a bug in $ORA_CRS_HOME/bin/racgvip. In this script change [[:alnum:]] to [a-z]. You may need to make alterations to this fix if your interfaces do not begin with a lowercase letter.
It might be simplest to reboot the nodes now to clear out any incorrect caching of virtual IP addresses.
The above applies for version 10.2.0.1 of CRS. Oracle rewrote the racgvip script for version 10.2.0.3, so the above patch is not required if you have applied patchset 2.
3.1.4. Testing
The status of the cluster can be queried with $ORA_CRS_HOME/bin/crs_stat -t. To test if clustered services start and stop, use $ORA_CRS_HOME/bin/srvctl start nodeapps -n docrac1, substituting start with stop and the relevant node names as appropriate. To simulate the loss of a node, run telinit 2 on that node. Running telinit 3 brings that node back into the cluster.
3.2. ASM
No problems encountered, but if ASM is installed before the cluster has been tested and stabilised, it will be necessary to rerun $ORACLE_HOME/bin/dbca and recreate the diskgroup(s) for the ASM instance.
3.3. Database
3.3.1. Installation
No problems encountered during the installation. The Enterprise Manager URL is created at http://docrac1:1158/em. Errors were reported from the Enterprise Manager console on the other RAC node as the database initially is not started.
As before, do not forget to edit the root.sh script to change the group nobody to nogroup before running.
Finally, remember the URLs that have been created for iSQL*Plus at http://docrac1:5560/isqlplus and iSQL*Plus DBA at http://docrac1:5560/isqlplus/dba.
3.3.2. Configuration
Add the necessary service aliases to your OID or LDAP server, i.e. (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=docrac1)(PORT=1521)))) for alias listener_docrac1, (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=docrac2)(PORT=1521)))) for alias listener_docrac2, and (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = docrac1-vip)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = docrac2-vip)(PORT = 1521)) (LOAD_BALANCE = yes) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = sales) (FAILOVER_MODE = (TYPE = SELECT) (METHOD = BASIC) (RETRIES = 180) (DELAY = 5) ) ) ) for transparent application failover for the database.
3.3.3. Testing
iSQL*Plus can be started by running $ORACLE_HOME/bin/isqlplusctl start. A database instance can be stopped by running $ORA_CRS_HOME/bin/srvctl stop instance -d sales -i sales1 and the database can be stopped by running $ORA_CRS_HOME/bin/srvctl stop database -d sales.