Recently I discovered that there are very few places where the process of adding new storage to ASM is documented. Many Oracle users evidently think that Oracle must be restarted in order to add storage to ASM. This is actually only true if it is necessary to restart the Oracle server or servers (in a cluster) in order for the new LUN to be recognized. On most servers these days, it is not necessary. These are the steps to adding a new LUN to ASM using ASMLIB.
These instructions assume we are working on a multi-node RAC system. They do not cover the steps required to get the LUN added to the operating system as that can vary by the server environment.
Have the storage or systems group allocate a new LUN to the RAC nodes. Ensure this is a single LUN visible to both nodes. If possible, It should appear in the same location on both servers, ie. if it is at /dev/sdi on node 1 it should show as /dev/sdi on node 2. The disk placement is not absolutely critical when using ASMLIB, but I still recommend it. If the disks are mounted in different paths on the two nodes it becomes much easier to make mistakes in managing the disks. If you do not use ASMLIB, you should use udev or other method of ensuring that the same disk always comes up in the same location, even after adding additional storage.
NOTE: Do not use thin provisioning to allocate this LUN!
- Create a single primary partition containing the entire LUN and make sure it is visible on both RAC nodes.
As root, on the first node of the cluster only, use the oracleasm program to allocate the disk to oracle. The command is ‘/etc/init.d/oracleasm createdisk <disk name> <path>’,
eg: ‘# /etc/init.d/oracleasm createdisk DATA03 /dev/sdi1’
Use oracleasm as oracle to verify that the disk is available: # /etc/init.d/oracleasm listdisks. The output should show all asm disks, including the new one:
oracle (+ASM2)$ /etc/init.d/oracleasm listdisks
On all other nodes of the cluster, as root, scan for the new oracle disk:
[root@kerbrac01 ~]# /etc/init.d/oracleasm scandisks
Scanning the system for Oracle ASMLib disks: [ OK ]
After running scandisk on the other nodes, run listdisks as oracle on each node to verify that the new disk is visible:
oracle (+ASM1)$ /etc/init.d/oracleasm listdisks
After verifying that the new disk is visible on all nodes, go back to the first node, connect as Oracle. Then set the environment to the ASM instance, and add the disk to the ASM diskgroup.
oracle (racdb1)$ . oraenv
ORACLE_SID = [racdb1] ? +ASM1
The Oracle base has been changed from /opt/oracle to /u01/oracle
oracle (+ASM1)$ sqlplus / as sysasm
SQL> alter diskgroup DATA01 add disk ‘DAT03’;
After the disk is added, ASM will balance data in the diskgroup across all the disks in the diskgroup so each disk has about the same amount of data. This operation is called rebalancing. The rebalance process can be viewed by select from the v$asm_operation view:
SQL> Select * from v$asm_operation;
GROUP_NUMBER OPERA STAT POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES ERROR_CODE
———— —– —- ———- ———- ———- ———- ———- ———– ——————————————–
1 REBAL RUN 5 5 267503 338682 5439 13
In order to speed up the process, increase the rebalance power. This setting defines how many resources the rebalance process will consume, the maximum value is 11 in oracle 10.2, and 1024 in Oracle 11. To increase the rebalance power, the command is ‘alter diskgroup <diskgroup name> rebalance power <power number>;, eg:
SQL> alter diskgroup data01 rebalance power 5;
- Once the rebalance is complete, the entire space added will be available. More space becomes available as the rebalance operation runs, so the entire space added will not be available immediately.
- As a general rule, it is not a good practice to add a new disk to ASM and drop a disk at the same time. If you are going to remove a LUN from ASM, wait for the first rebalance operation to complete, then drop the old disk, and verify that the rebalance operation is complete before attempting further operations.