ASMLIB in OEL 7

In my previous port I wrote about using UDEV rules for device name persistence in an Oracle RAC environment. In this post, I will discuss using ASMLIB to the same purpose. I had intended to also talk about using kmod-oracleasm, which is the equivalent pack to ASMLIB, but unique to RHEL. However, due to licensing issues I was not able to create a set of RHEL VMware workstations to this purpose

AMLIB is the Oracle preferred solution to device name persistence. ASMLIB will write an ASM header to each physical device, as opposed to UDEV, which uses the existing SCSI unique ID to map the device name. In my opinion, ASMLIB is superior to UDEV for device name persistence.

In this post, I will also include the steps to get the iSCSI devices mapped into the OEL operating system. They were not included in my previous posts for the purpose of brevity. These steps are identical to those required in the previous post on UDEV rules. These are primarily included for those who wish to set up their own home cluster. For a true commercial level cluster, use the licensed version of the iscsi target, or other commercial software. For instructions on how to set up the ISCSI target within Openfiler, there are several good sets of instructions available, one is here: http://techhead.co/how-to-configure-openfiler-v23-iscsi-storage-for-use-with-vmware-esx/.

The first step is to turn on the iSCSI initiator on the operating system. The initiator appears to be part of the default kernel installation. The commands to activate it are here:

service iscsid start

chkconfig iscsid on

 

In OEL 7, the commands will be translated to systemctl, so the commands you see will be this:

[root@oel7641 ~]# service iscsid start

Redirecting to /bin/systemctl start iscsid.service

[root@oel7641 ~]# chkconfig iscsid on

Note: Forwarding request to ‘systemctl enable iscsid.service’.

[root@oel7641 ~]#

 

In order to map the openfiler iSCSI devices to the Linux operating system, we use the sendtargets option of the iscsiadm command, then we use the login option to login to the devices. After this is done, the iscsi devices will appear as standard Linux scsi devices:

[root@oel7641-localdomain ~]# iscsiadm -m discovery -t sendtargets -p 192.168.213.100:3260

192.168.213.100:3260,1 iqn.2006-01.com.openfiler:tsn.70d7d9301f39

[root@oel7641-localdomain ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:tsn.70d7d9301f39 -p 192.168.213.100:3260 –login

Logging in to [iface: default, target: iqn.2006-01.com.openfiler:tsn.70d7d9301f39, portal: 192.168.213.100,3260] (multiple)

Login to [iface: default, target: iqn.2006-01.com.openfiler:tsn.70d7d9301f39, portal: 192.168.213.100,3260] successful.

 

In the output below, devices sdb through sdg are the iscsi devices:

 

[root@oel7641 ~]# fdisk -l

 

Disk /dev/sda: 161.1 GB, 161061273600 bytes, 314572800 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk label type: dos

Disk identifier: 0x0003b5aa

 

Device Boot Start End Blocks Id System

/dev/sda1 * 2048 1026047 512000 83 Linux

/dev/sda2 1026048 314572799 156773376 8e Linux LVM

 

Disk /dev/mapper/ol_oel764-swap: 16.9 GB, 16852713472 bytes, 32915456 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

 

Disk /dev/mapper/ol_oel764-root: 87.2 GB, 87241523200 bytes, 170393600 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

 

Disk /dev/mapper/ol_oel764-home: 56.4 GB, 56438554624 bytes, 110231552 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

 

Disk /dev/sdb: 8187 MB, 8187281408 bytes, 15990784 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk label type: dos

Disk identifier: 0x2adfadeb

 

Disk /dev/sdc: 8187 MB, 8187281408 bytes, 15990784 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk label type: dos

Disk identifier: 0x536c247a

 

Disk /dev/sdd: 8187 MB, 8187281408 bytes, 15990784 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk label type: dos

Disk identifier: 0xcf43d9a8

 

Disk /dev/sde: 40.9 GB, 40936407040 bytes, 79953920 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk label type: dos

Disk identifier: 0x253975b8

 

Disk /dev/sdf: 40.9 GB, 40936407040 bytes, 79953920 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk label type: dos

Disk identifier: 0xbdab400b

 

Disk /dev/sdg: 40.9 GB, 40936407040 bytes, 79953920 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk label type: dos

Disk identifier: 0x85ddc3a3

 

At this point, we want to make sure that the devices are also available on the second node. The simplest method to do this is to copy the subdirectories under /var/lib/iscsi to the second and additional nodes in the cluster, then run the commands to start iscsi on those nodes.

scp –r /var/lib/iscsi oel7642:/var/lib/scsi

(log into oel7642)

[root@oel7642 ~]# service iscsid start

Redirecting to /bin/systemctl start iscsid.service

[root@oel7642 ~]# chkconfig iscsid on

Note: Forwarding request to ‘systemctl enable iscsid.service’.

[root@oel7642 ~]#exit

 

Next, we need to partition the iscsi devices. Also, since these are VMware vmdk’s, we need to make sure that the partitions are block aligned for best performance. Here is a typical fdisk conversation to accomplish that purpose. Note that we only partition and run oracleasm on a single node. After the devices are marked for ASM, we just scan for them on the second node and additional nodes:

 

[root@oel7641 ~]# fdisk /dev/sdb

Welcome to fdisk (util-linux 2.23.2).

 

Changes will remain in memory only, until you decide to write them.

Be careful before using the write command.

 

Device does not contain a recognized partition table

Building a new DOS disklabel with disk identifier 0x2adfadeb.

 

Command (m for help): n

Partition type:

p primary (0 primary, 0 extended, 4 free)

e extended

Select (default p): p

Partition number (1-4, default 1): 1

First sector (2048-15990783, default 2048):

Using default value 2048

Last sector, +sectors or +size{K,M,G} (2048-15990783, default 15990783):

Using default value 15990783

Partition 1 of type Linux and of size 7.6 GiB is set

 

Command (m for help): t

Selected partition 1

Hex code (type L to list all codes): fb

Changed type of partition ‘Linux’ to ‘VMware VMFS’

 

Command (m for help): x

 

Expert command (m for help): b

Selected partition 1

New beginning of data (1-15990783, default 2048): 2048

 

Expert command (m for help): w

The partition table has been altered!

 

Calling ioctl() to re-read partition table.

Syncing disks.

 

Because of the number of devices, I created a very simple script to run the fdisk conversation on the remainder of the devices, I called it doit.sh. Don’t complain about the naming. This script assumes we want to use the entire device for ASM storage.

Here is the script:

[root@oel7641 ~]# cat doit.sh

#!/bin/sh
fdisk $1 <<EOF
n
p
1

 
 t
fb
x
b
2048
w
exit
EOF

 

The script is called with the script name followed by the device name:

[root@oel7641 ~]# ./doit.sh /dev/sdc

[root@oel7641 ~]# ./doit.sh /dev/sdd

[root@oel7641 ~]# ./doit.sh /dev/sde

[root@oel7641 ~]# ./doit.sh /dev/sdf

[root@oel7641 ~]# ./doit.sh /dev/sdg

 

At that point, all the device are partitioned and it is time to create the ASM devices.

 

So next, we need to install ASM. It is not installed by default (currently) in the OEL7 Kernel. We use yum to install ORACLEASM, two packages are required ORACLEASM and ORACLEASM-SUPPORT:

 

[root@oel7641-localdomain ~]# yum install oracleasm

Loaded plugins: langpacks, rhnplugin

This system is receiving updates from ULN.

ol7_x86_64_Dtrace_userspace | 1.2 kB 00:00:00

ol7_x86_64_UEKR3 | 1.2 kB 00:00:00

ol7_x86_64_addons | 1.2 kB 00:00:00

ol7_x86_64_latest | 1.4 kB 00:00:00

ol7_x86_64_latest/updateinfo | 33 kB 00:00:00

ol7_x86_64_latest/primary | 4.5 MB 00:00:02

ol7_x86_64_latest 6631/6631

ol7_x86_64_optional_latest | 1.2 kB 00:00:00

ol7_x86_64_optional_latest/updateinfo | 32 kB 00:00:00

ol7_x86_64_optional_latest/primary | 1.4 MB 00:00:00

ol7_x86_64_optional_latest 5022/5022

Resolving Dependencies

–> Running transaction check

—> Package kernel-uek.x86_64 0:3.8.13-35.3.4.el7uek will be installed

–> Processing Dependency: kernel-firmware = 3.8.13-35.3.4.el7uek for package: kernel-uek-3.8.13-35.3.4.el7uek.x86_64

–> Running transaction check

—> Package kernel-uek-firmware.noarch 0:3.8.13-35.3.4.el7uek will be installed

–> Finished Dependency Resolution

 

Dependencies Resolved

 

================================================================================================================================================================

Package Arch Version Repository Size

================================================================================================================================================================

Installing:

kernel-uek x86_64 3.8.13-35.3.4.el7uek ol7_x86_64_UEKR3 31 M

kernel-uek-firmware noarch 3.8.13-35.3.4.el7uek ol7_x86_64_UEKR3 1.8 M

 

Transaction Summary

================================================================================================================================================================

Install 2 Packages

 

Total size: 33 M

Total download size: 1.8 M

Installed size: 112 M

Is this ok [y/d/N]: y

Downloading packages:

No Presto metadata available for ol7_x86_64_UEKR3

kernel-uek-firmware-3.8.13-35.3.4.el7uek.noarch.rpm | 1.8 MB 00:00:01

Running transaction check

Running transaction test

Transaction test succeeded

Running transaction

Installing : kernel-uek-firmware-3.8.13-35.3.4.el7uek.noarch 1/2

Installing : kernel-uek-3.8.13-35.3.4.el7uek.x86_64 2/2

Verifying : kernel-uek-firmware-3.8.13-35.3.4.el7uek.noarch 1/2

Verifying : kernel-uek-3.8.13-35.3.4.el7uek.x86_64 2/2

 

Installed:

kernel-uek.x86_64 0:3.8.13-35.3.4.el7uek kernel-uek-firmware.noarch 0:3.8.13-35.3.4.el7uek

 

Complete!

 

[root@oel7641-localdomain ~]# yum install oracleasm-support

Loaded plugins: langpacks, rhnplugin

This system is receiving updates from ULN.

Resolving Dependencies

–> Running transaction check

—> Package oracleasm-support.x86_64 0:2.1.8-3.el7 will be installed

–> Finished Dependency Resolution

 

Dependencies Resolved

 

================================================================================================================================================================

Package Arch Version Repository Size

================================================================================================================================================================

Installing:

oracleasm-support x86_64 2.1.8-3.el7 ol7_x86_64_latest 79 k

 

Transaction Summary

================================================================================================================================================================

Install 1 Package

 

Total download size: 79 k

Installed size: 242 k

Is this ok [y/d/N]: y

Downloading packages:

oracleasm-support-2.1.8-3.el7.x86_64.rpm | 79 kB 00:00:00

Running transaction check

Running transaction test

Transaction test succeeded

Running transaction

Installing : oracleasm-support-2.1.8-3.el7.x86_64 1/1

Note: Forwarding request to ‘systemctl enable oracleasm.service’.

ln -s ‘/usr/lib/systemd/system/oracleasm.service’ ‘/etc/systemd/system/multi-user.target.wants/oracleasm.service’

Verifying : oracleasm-support-2.1.8-3.el7.x86_64 1/1

 

Installed:

oracleasm-support.x86_64 0:2.1.8-3.el7

 

Complete!

 

 

Once the ASMLIB is installed, it must be configured. This just amounts to telling Linux the owner of the ASM disks, and the location of the disks, along with some minor options.

 

[root@oel7641-localdomain ~]# /etc/init.d/oracleasm configure

Configuring the Oracle ASM library driver.

 

This will configure the on-boot properties of the Oracle ASM library

driver. The following questions will determine whether the driver is

loaded on boot and what permissions it will have. The current values

will be shown in brackets (‘[]’). Hitting <ENTER> without typing an

answer will keep that current value. Ctrl-C will abort.

 

Default user to own the driver interface []: oracle

Default group to own the driver interface []: dba

Scan for Oracle ASM disks on boot (y/n) [y]:

Writing Oracle ASM library driver configuration: done

Initializing the Oracle ASMLib driver: [ OK ]

Scanning the system for Oracle ASMLib disks: [ OK ]

 

With ASM configured, we mark the disks for ASM. Note that unlike UDEV this puts an actual label on the disk, it does not use the existing UUID to identify the disk.

 

[root@oel7641 ~]# /etc/init.d/oracleasm createdisk asmdisk1 /dev/sdb1

Marking disk “asmdisk1” as an ASM disk: [ OK ]

[root@oel7641 ~]# /etc/init.d/oracleasm createdisk asmdisk2 /dev/sdc1

Marking “asmdisk2” as an ASM disk:

[root@oel7641 ~]# /etc/init.d/oracleasm createdisk asmdisk3 /dev/sdd1

Marking disk “asmdisk3” as an ASM disk: [ OK ]

[root@oel7641 ~]# /etc/init.d/oracleasm createdisk asmdisk4 /dev/sde1

Marking disk “asmdisk4” as an ASM disk: [ OK ]

[root@oel7641 ~]# /etc/init.d/oracleasm createdisk asmdisk5 /dev/sdf1

Marking disk “asmdisk5” as an ASM disk: [ OK ]

[root@oel7641 ~]# /etc/init.d/oracleasm createdisk asmdisk6 /dev/sdg1

Marking disk “asmdisk6” as an ASM disk: [ OK ]

[root@oel7641 ~]# /etc/init.d/oracleasm listdisks

ASMDISK1

ASMDISK2

ASMDISK3

ASMDISK4

ASMDISK5

ASMDISK6

 

Next, we make sure that the other cluster nodes can see the ASM disks. We do not need to reboot the second node of the cluster, nor do we run the createdisk command on the other nodes in the cluster, we scan for the storage devices on the other nodes, to make oracleasm aware of the disks and we are done:

 

[root@oel7642 ~]# /etc/init.d/oracleasm scandisks

Scanning the system for Oracle ASMLib disks: [ OK ]

[root@oel7642 ~]# /etc/init.d/oracleasm listdisks

ASMDISK1

ASMDISK2

ASMDISK3

ASMDISK4

ASMDISK5

ASMDISK6

[root@oel7642 ~]#

 

If you want to verify that the disks appear properly reboot both servers and check.

 

At this point, the Oracle Grid Infrastructure for a cluster can be installed. The ASM_DISKSTRING for the installation will be /dev/oracleasm/disks/*.

 

According to Oracle documentation, starting with Oracle version 12, there is a piece of software called the ASM Filter Driver (ASMFD) that prevents non-oracle software from writing to Oracle data files. I assume that by this they mean software not owned by the Oracle software owner. In any case, the ASMFD replaces ASMLIB. In my next post, I will go over the steps to replace ASMLIB with ASMFD.

 

 

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: