Udev Rules for Oracle Storage

As I have discussed in the past, one of the problems that must be dealt with when configuring Oracle RAC is device naming persistence. In order for RAC to work properly, the storage device names must be the same across all servers.

There are two ways to deal with this, the Oracle supplied asmlib (now ASM filter driver, asmfd), and udev rules.  Asmlib is supplied with Oracle Enterprise Linux, and also now with RHEL.  However, many people prefer not to deal with the complications involved with asmlib or asmfd, and use udev rules.

Remember when using udev rules, you must put the same rules file on all the servers in the cluster, most often the file is something like this: /etc/udev/rules.d/99-oracleasm.rules.

With the latest releases of RHEL, udev naming has changed in, this time for the better. The only problem of course is that as per usual, its poorly documented. In the past, we have used lsscsi for device names,  in conjunction with scsi_id. But now there is a better way. After substantial experimentation trying to get lsscsi to work, I did some extensive research and finally determined that the best way to set up udev rules for your oracle storage is to use udevadm. This method should actually work for almost every storage supplier, so this is very handy. The command is udevadm. In our case, udevadm info –query=propery –name <your device name>.

This returns lots of useful information:

udevadm info –query=property –name /dev/sdb

DEVLINKS=/dev/disk/by-id/scsi-36000c2984bb594f77534b298654126f3 /dev/disk/by-id/wwn-0x6000c2984bb594f77534b298654126f3 /dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:1:0 
DEVNAME=/dev/sdb 
DEVPATH=/devices/pci0000:00/0000:00:10.0/host0/target0:0:1/0:0:1:0/block/sdb 
DEVTYPE=disk 
FC_TARGET_LUN=0 
ID_BUS=scsi 
ID_MODEL=VMware_Virtual_S 
ID_MODEL_ENC=VMware\x20Virtual\x20S 
ID_PATH=pci-0000:00:10.0-scsi-0:0:1:0 
ID_PATH_TAG=pci-0000_00_10_0-scsi-0_0_1_0 
ID_REVISION=1.0 
ID_SCSI=1 
ID_SCSI_SERIAL=6000c2984bb594f77534b298654126f3 
ID_SERIAL=36000c2984bb594f77534b298654126f3 
ID_SERIAL_SHORT=6000c2984bb594f77534b298654126f3 
ID_TYPE=disk 
ID_VENDOR=VMware_ 
ID_VENDOR_ENC=VMware\x2c\x20 
ID_WWN=0x6000c2984bb594f7 
ID_WWN_VENDOR_EXTENSION=0x7534b298654126f3 
ID_WWN_WITH_EXTENSION=0x6000c2984bb594f77534b298654126f3 
MAJOR=8 
MINOR=16 
MPATH_SBIN_PATH=/sbin 
SUBSYSTEM=block 
TAGS=:systemd: 
USEC_INITIALIZED=66016

In our case, we are interested in any of the three highlighted serial numbers. Any of these can be used and we have chosen to use the ID_SERIAL_SHORT, the command in your rules file is: ENV{ID_SERIAL_SHORT} The line in the rules file is:

KERNEL==”sd?”, ENV{ID_SERIAL}==”36000c2984bb594f77534b298654126f3″ SYMLINK+=”oracleasm/asm-disk1″, OWNER=”oracle”, GROUP=”dba”, MODE=”0660″

This will give a consistent name to the storage device for your oracle database.

After a little experimentation, I wrote this simple script to auto-generate the rules:

#!/bin/sh 
COUNTER=0 
for disk in `ls /dev/sd* -c1 | grep -v sda` 
do 
shortserial=`udevadm info --query=property --name $disk | grep ID_SERIAL_SHORT | awk -F "=" '{print $2}'` 
COUNTER=$((COUNTER+1)) 
echo KERNEL==\"sd?\", ENV{ID_SERIAL_SHORT}==\"$shortserial\" SYMLINK+=\"oracleasm/asm-disk$COUNTER\", OWNER=\"oracle\", GROUP=\"dba\", MODE=\"0660\" 
done

Note that in initialization of the ‘for’ loop, we do grep -v sda, this is because sda is almost always going to be an internal device, not attached storage, and not used for the database storage. This the above script produces output like this:

KERNEL=="sd?", ENV{ID_SERIAL_SHORT}=="6000c291476a173c2886cc425ea7050e" SYMLINK+="oracleasm/asm-disk1", OWNER="oracle", GROUP="dba", MODE="0660" 
KERNEL=="sd?", ENV{ID_SERIAL_SHORT}=="6000c29ef4eff651c416774c6172fe33" SYMLINK+="oracleasm/asm-disk2", OWNER="oracle", GROUP="dba", MODE="0660" 
KERNEL=="sd?", ENV{ID_SERIAL_SHORT}=="6000c2994337b652f937a6ffe64f9217" SYMLINK+="oracleasm/asm-disk3", OWNER="oracle", GROUP="dba", MODE="0660" 
KERNEL=="sd?", ENV{ID_SERIAL_SHORT}=="6000c29f2971757fc3fb8b34e5e9e35f" SYMLINK+="oracleasm/asm-disk4", OWNER="oracle", GROUP="dba", MODE="0660" 
KERNEL=="sd?", ENV{ID_SERIAL_SHORT}=="6000c2907f869d72b62d5774f933a97c" SYMLINK+="oracleasm/asm-disk5", OWNER="oracle", GROUP="dba", MODE="0660" 
KERNEL=="sd?", ENV{ID_SERIAL_SHORT}=="6000c29ae1d265baf6604af619b2f79a" SYMLINK+="oracleasm/asm-disk6", OWNER="oracle", GROUP="dba", MODE="0660" 
KERNEL=="sd?", ENV{ID_SERIAL_SHORT}=="6000c299e07c9ed590a2884cabfa855e" SYMLINK+="oracleasm/asm-disk7", OWNER="oracle", GROUP="dba", MODE="0660" 
KERNEL=="sd?", ENV{ID_SERIAL_SHORT}=="6000c29e1848051e29a84408a27d2522" SYMLINK+="oracleasm/asm-disk8", OWNER="oracle", GROUP="dba", MODE="0660" 
KERNEL=="sd?", ENV{ID_SERIAL_SHORT}=="6000c290af449698eb820b0e4b329471" SYMLINK+="oracleasm/asm-disk9", OWNER="oracle", GROUP="dba", MODE="0660" 
KERNEL=="sd?", ENV{ID_SERIAL_SHORT}=="6000c2997873b2d7c86a1d561bedb294" SYMLINK+="oracleasm/asm-disk10", OWNER="oracle", GROUP="dba", MODE="0660" 
KERNEL=="sd?", ENV{ID_SERIAL_SHORT}=="6000c2984bb594f77534b298654126f3" SYMLINK+="oracleasm/asm-disk11", OWNER="oracle", GROUP="dba", MODE="0660"

We copy the output above to the file /etc/udev/rules.d/99-oracleasm.rules on all the servers in the cluster, and run the command ‘udevadm -trigger’ or reboot all nodes.  All the storage devices will then come up in the same location, and typically the Oracle installer will find them when setting up the Oracle Grid Infrastructure.

In summary, we have successfully generated udev rules for attached storage. The script provided is very simple, and should work for all of the attached storage devices.

4 Responses to “Udev Rules for Oracle Storage”

  1. Yogi Says:

    Is there a specific reason, for still using regex in device name while it is being identified by serial i’d?
    Do you anticipate device name change? If yes, how do you handle this in ASM?

    -Yogi

    • dbakerber Says:

      The point of using udev rules is to relate specific serial numbers to specific named devices, eg /dev/oracleasm/asm-disk1, etc. The devices show up in /dev originally as /dev/sdb throught /dev/sdxx. There is no guarantee that the /dev/sd location will persist across system reboots, so udev is used to ensure that each device serial number is always assigned to the same asm-disk number, regardless of the /dev/sd name that is assigned at boot. The /dev/sda device will always be an internal device, but beyond that they could end up at any location.

      • Yogi Says:

        This is interesting.
        In my present org, VMware and OS team has assured me that device names will never change, unless we rebuild OS itself. So far it has worked for more than half a decade.

        I fear, ASM devices can be handled with your workaround, devices that are part of volume groups that we consume for various other filesystems on the box, stand a big risk across reboots.
        How do you handle those? We don’t keep anything in sda other than root and basic OS filesystems. Everything else gets a separate device(s) to build upon.

        -Yogi

      • dbakerber Says:

        I am not sure what you are asking. This isnt really a workaround, udev rules are the standard method for naming devices on linux, so it will work for your other devices also. Note that its only used for identify physical devices or storage luns, not file systems.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s


%d bloggers like this: