Skip to main content

1. Introduction

This brief HOWTO is based on the excellent article from Oracle detailing how to build a RAC cluster on GNU/Linux using ASM as the storage medium. It also pulls some information from this other great article though I used an iSCSI SAN for shared storage.

It is assumed that the Etch distribution of Debian GNU/Linux is used. The Linux kernel that this was tested on was version 2.6.18-4-686.

2. Working around assumptions

2.1. Packages

You will need some extra packages that may not have been installed. For the Oracle Universal Installer you will need to install libxp6, libstdc++5, unzip and build-essential. For CRS you will need arping and iproute, and for the database you will need to install libaio1 as well.

2.2. File paths

The installer assumes that awk and basename are installed in /bin rather than /usr/bin. CRS, specifically $ORA_CRS_HOME/bin/racgvip, assumes that sort is also in /bin. Therefore, create soft links, i.e.

ln -s /usr/bin/awk /bin
ln -s /usr/bin/basename /bin
ln -s /usr/bin/sort /bin

If you find that your virtual host IP addresses keep bouncing from node to node then you need to create the soft link for sort as detailed above.

2.3. Init

The guide assumes that the system will be at run level 3, but Debian GNU/Linux runs at run level 2. Change id:2:initdefault: in /etc/inittab to id:3:initdefault: and switch run levels by issuing telinit 3.

2.4. Kernel modules

The raw kernel module is not loaded by default, so add raw to /etc/modules and run modprobe raw to load it immediately.

Udev only creates the /dev/rawctl device node, so a script is required to create the remaining device nodes that act as the raw devices. The following script, named /etc/init.d/rawdevices will suffice for this purpose:

#! /bin/sh
set -e

# /etc/init.d/rawdevices: start and stop the raw(8) devices

test -x /sbin/raw || exit 0

export PATH="${PATH:+$PATH:}/usr/sbin:/sbin"

case "$1" in
        echo -n "Starting RAW devices: "

        mkdir -p /dev/raw
        for i in 1 2 3 4 5 6 7 8 9 10; do
                mknod /dev/raw/raw$i c 162 $i
                chown root:oinstall /dev/raw/raw$i
                chmod 660 /dev/raw/raw$i

        # OCR devices
        raw /dev/raw/raw1 /dev/sda1
        raw /dev/raw/raw2 /dev/sdb1
        # Voting disk devices
        raw /dev/raw/raw3 /dev/sdc1
        raw /dev/raw/raw4 /dev/sdd1
        raw /dev/raw/raw5 /dev/sde1
        # ASM disk devices
        raw /dev/raw/raw6 /dev/sda2
        raw /dev/raw/raw7 /dev/sdb2
        raw /dev/raw/raw8 /dev/sdc2
        raw /dev/raw/raw9 /dev/sdd2
        raw /dev/raw/raw10 /dev/sde2

        echo "rawdevices."
        echo -n "Stopping RAW devices: "
        for i in 1 2 3 4 5 6 7 8 9 10; do
                raw /dev/raw/raw$i 0 0
                rm -f /dev/raw/raw$i
        rmdir /dev/raw

        echo "rawdevices."

        echo "Usage: /etc/init.d/rawdevices {start|stop}"
        exit 1

exit 0

Install the script with update-rc.d rawdevices defaults and start it with /etc/init.d/rawdevices start. It is not sufficient to call the standard MAKEDEV raw script as that script checks if Udev is running and refuses to create the device nodes in /dev if it is.

2.5. Root scripts

The Oracle scripts must be all altered before running. Any references to chgrp nobody must be changed to chgrp nogroup.

3. Installation

3.1. CRS

3.1.1. Startup

The cluster scripts are not set to start on boot properly. Run update-rc.d start 96 3 4 5 . stop 96 0 1 2 6 . to resolve this. This will also allow you to shut the cluster down on demand by changing to run level 2, and restart it by changing back to run level 3.

3.1.2. Virtual node IP addresses

There may be issues installing CRS because the virtual node IP addresses are in the private subnet ranges. To override this, run $ORA_CRS_HOME/bin/vipca as the root user on any affected nodes.

3.1.3. Virtual node IP network interfaces

Oracle CRS virtual node IP addresses use IPv4, and so CRS must create a network interface alias to assign more than one IP address to a network interface. If you use eth0 for your public interface, you would expect to see interface eth0:0 holding the virtual IP. Unfortunately, you will see many more interface aliases than this. This is due to a bug in $ORA_CRS_HOME/bin/racgvip. In this script change [[:alnum:]] to [a-z]. You may need to make alterations to this fix if your interfaces do not begin with a lowercase letter.

It might be simplest to reboot the nodes now to clear out any incorrect caching of virtual IP addresses.

The above applies for version of CRS. Oracle rewrote the racgvip script for version, so the above patch is not required if you have applied patchset 2.

3.1.4. Testing

The status of the cluster can be queried with $ORA_CRS_HOME/bin/crs_stat -t. To test if clustered services start and stop, use $ORA_CRS_HOME/bin/srvctl start nodeapps -n docrac1, substituting start with stop and the relevant node names as appropriate. To simulate the loss of a node, run telinit 2 on that node. Running telinit 3 brings that node back into the cluster.

3.2. ASM

No problems encountered, but if ASM is installed before the cluster has been tested and stabilised, it will be necessary to rerun $ORACLE_HOME/bin/dbca and recreate the diskgroup(s) for the ASM instance.

3.3. Database

3.3.1. Installation

No problems encountered during the installation. The Enterprise Manager URL is created at http://docrac1:1158/em. Errors were reported from the Enterprise Manager console on the other RAC node as the database initially is not started.

As before, do not forget to edit the script to change the group nobody to nogroup before running.

Finally, remember the URLs that have been created for iSQL*Plus at http://docrac1:5560/isqlplus and iSQL*Plus DBA at http://docrac1:5560/isqlplus/dba.

3.3.2. Configuration

Add the necessary service aliases to your OID or LDAP server, i.e. (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=docrac1)(PORT=1521)))) for alias listener_docrac1, (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=docrac2)(PORT=1521)))) for alias listener_docrac2, and (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = docrac1-vip)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = docrac2-vip)(PORT = 1521)) (LOAD_BALANCE = yes) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = sales) (FAILOVER_MODE = (TYPE = SELECT) (METHOD = BASIC) (RETRIES = 180) (DELAY = 5) ) ) ) for transparent application failover for the database.

3.3.3. Testing

iSQL*Plus can be started by running $ORACLE_HOME/bin/isqlplusctl start. A database instance can be stopped by running $ORA_CRS_HOME/bin/srvctl stop instance -d sales -i sales1 and the database can be stopped by running $ORA_CRS_HOME/bin/srvctl stop database -d sales.

See also