ASMDisks

From Outrun Wiki
Jump to: navigation, search

Intro

ASMdisks provides a way to manage Oracle ASM volumes using Linux UDEV as alternative to Oracle ASMLib.

A deeper explanation here: Fun with Linux UDEV and ASM: Using UDEV to create ASM disk volumes

You can use a combination of entire disks, partitions or mapper (including multipath) devices to configure as block devices for Oracle ASM (or other purposes such as destructive IOPS performance testing).

The asmdisks package provides some scripts of which /usr/bin/asm is the most important. It manages a file named /etc/asmtab with definitions for configuring Linux UDEV devices so you don't have to manually create the udev rules file. It also provides consistent naming based on SCSI serial IDs or device mapper configuration. By default, managed volumes appear as /dev/oracleasm/<volname> with owner "grid" and group "asmdba". In Oracle ASM, point the ASM_DISKSTRING to /dev/oracleasm to make things come to life.

asmdisks can handle raw volumes (i.e. /dev/sdi), partitions (/dev/sdi1) or mapper volumes (/dev/mapper/datavg-vol1) including Linux Multipath devices. Support for EMC Powerpath and EMC ScaleIO devices is on my todo list.


Installation

  1. Set up the Outrun Extras YUM repository:
    yum install http://yum.outrun.nl/outrun-extras.rpm
  2. Install the "asmdisks" package:
    yum install asmdisks
  3. If you run VMware, add
    disk.EnableUUID=true
    to the VMX configuration (see scsi_id on CentOS 6 on VMware returns null) and reboot

Because asmdisks is part of the Outrun Extras YUM repository on my web server, you can get updates via "yum update" as soon as I publish a new version of the package. In case this leads to problems, you can do "yum downgrade asmdisks" to go back to the previous release. The source RPM is also available from my web server if you would like to rebuild the package for your own purpose - it's licensed under GPLv3 so you're free to modify it in any way you like. If you make useful improvements I would appreciate to include these in the package (will list your name under authors if you like).

Info and manual

man asm man asmstat less /usr/share/doc/asmdisks/README

ASMdisks description

If you have asmdisks installed as well as bash-completion, then you can use TAB completion. The "asm" command also responds to -? or -h options for help, or just "asm" without options. There's a man page ("man asm") as well with more detailed usage info.

TAB completion shows available commands and a list of available devices or volumes where you need them.

Showing available disks

# asm disks
 /dev/sda    [2:0:0:0]  20.00 GB partitioned
 /dev/sdb    [2:0:1:0]   4.00 GB available
 /dev/sdc    [2:0:2:0]   4.00 GB available
 /dev/sdd    [2:0:3:0]   1.00 GB available
 /dev/sde    [2:0:4:0]  20.00 GB available
 /dev/sdf    [2:0:8:0]   2.00 GB available
 /dev/sdg    [2:0:9:0]   2.00 GB available
 /dev/sdh    [3:0:8:0]   2.00 GB available
 /dev/sdi    [3:0:9:0]   2.00 GB available
 /dev/sdj    [4:0:8:0]   2.00 GB available
 /dev/sdk    [4:0:9:0]   2.00 GB available
 /dev/sdl    [5:0:8:0]   2.00 GB available
 /dev/sdm    [5:0:9:0]   2.00 GB available
 /dev/sdn   [5:0:14:0]  16.00 GB available

Adding disks

# asm createdisk vol1 /dev/sdb
# asm disks
/dev/sda    [2:0:0:0]  20.00 GB partitioned 
/dev/sdb    [2:0:1:0]   4.00 GB configured as /dev/oracleasm/vol1 

Note that TAB completion works [1]: try "asm [TAB][TAB] c [TAB] vol2 [TAB][TAB][TAB] and you will see something like this:

# asm createdisk vol2 /dev/
/dev/mapper/rvg-home  /dev/mapper/rvg-tmp   /dev/sdd              
/dev/sdh              /dev/sdl               /dev/mapper/rvg-opt   /dev/mapper/rvg-usr   
/dev/sde              /dev/sdi              /dev/sdm               /dev/mapper/rvg-root  
/dev/mapper/rvg-var   /dev/sdf              /dev/sdj              /dev/sdn               
/dev/mapper/rvg-swap  /dev/sdc              /dev/sdg              /dev/sdk               
# asm createdisk vol2 /dev/sdc

Listing volumes

# asm list
vol1 4.00 GB [-] sdb
vol2 4.00 GB [-] sdc

Here the volume name is listed, the size, mapper name (not set here so it shows [-]), linux device name.

Showing ASMdisks volumes under Linux

The default diskstring is /dev/oracleasm so the shorthand vol1 and vol2 here actually means /dev/oracleasm/vol1 and ../vol2:

# ls -ald /dev/oracleasm/*
brw-rw---- 1 grid asmdba 8, 16 nov  7 16:06 /dev/oracleasm/vol1
brw-rw---- 1 grid asmdba 8, 32 nov  7 16:06 /dev/oracleasm/vol2

Using other disk strings

What if we want another disk string? Just enter it as a prefix to the volume name:

# asm createdisk mypath/vol3 /dev/sdd 
# asm disks
/dev/sda    [2:0:0:0]  20.00 GB partitioned 
/dev/sdb    [2:0:1:0]   4.00 GB configured as /dev/oracleasm/vol1 
/dev/sdc    [2:0:2:0]   4.00 GB configured as /dev/oracleasm/vol2 
/dev/sdd    [2:0:3:0]   1.00 GB configured as /dev/mypath/vol3 
# asm list
mypath/vol3              1.00 GB                  [-] sdd                      
vol1                     4.00 GB                  [-] sdb                      
vol2                     4.00 GB                  [-] sdc             

Using disk partitions

Some people prefer to use disk partitions instead of whole disks. Make sure your disk partitions are aligned in that case (at 1 MiB here):

# parted /dev/sde mklabel msdos
Information: You may need to update /etc/fstab.                           
# parted /dev/sde mkpart primary 1m 100%
Information: You may need to update /etc/fstab.         
# asm disks
/dev/sda    [2:0:0:0]  20.00 GB partitioned 
/dev/sdb    [2:0:1:0]   4.00 GB configured as /dev/oracleasm/vol1 
/dev/sdc    [2:0:2:0]   4.00 GB configured as /dev/oracleasm/vol2 
/dev/sdd    [2:0:3:0]   1.00 GB configured as /dev/mypath/vol3 
/dev/sde    [2:0:4:0]  20.00 GB partitioned 
# asm createdisk vol4 /dev/sde1
# asm disks
/dev/sda    [2:0:0:0]  20.00 GB partitioned 
/dev/sdb    [2:0:1:0]   4.00 GB configured as /dev/oracleasm/vol1 
/dev/sdc    [2:0:2:0]   4.00 GB configured as /dev/oracleasm/vol2 
/dev/sdd    [2:0:3:0]   1.00 GB configured as /dev/mypath/vol3 
/dev/sde    [2:0:4:0]  20.00 GB partitioned 

Note that /dev/sde1 does not show up under disks as sde might have more than one partition. It shows up if we list by volume:

# asm list
mypath/vol3              1.00 GB                  [-] sdd                      
vol1                     4.00 GB                  [-] sdb                      
vol2                     4.00 GB                  [-] sdc                      
vol4                    19.99 GB                  [-] sde1              

Using LVM logical volumes as ASM devices

For small test servers you might want to test with multiple ASM volumes mapping to logical volumes in a Linux LVM:

# vgcreate -Ay asmdata /dev/sdg /dev/sdh
  Physical volume "/dev/sdg" successfully created
  Physical volume "/dev/sdh" successfully created
  Volume group "asmdata" successfully created
# lvcreate -Ay -L1G -nvol6 asmdata 
  Logical volume "vol6" created
# lvcreate -Ay -L1G -nvol7 asmdata 
  Logical volume "vol7" created
# lvcreate -Ay -L1G -nvol8 asmdata 
  Logical volume "vol8" created
# ls -al /dev/asmdata/
total 0
drwxr-xr-x  2 root root  100 nov  7 16:19 .
drwxr-xr-x 19 root root 4100 nov  7 16:19 ..
lrwxrwxrwx  1 root root    7 nov  7 16:19 vol6 -> ../dm-7
lrwxrwxrwx  1 root root    7 nov  7 16:19 vol7 -> ../dm-8
lrwxrwxrwx  1 root root    7 nov  7 16:19 vol8 -> ../dm-9
# ls -al /dev/mapper/asmdata-vol*
lrwxrwxrwx 1 root root 7 nov  7 16:19 /dev/mapper/asmdata-vol6 -> ../dm-7
lrwxrwxrwx 1 root root 7 nov  7 16:19 /dev/mapper/asmdata-vol7 -> ../dm-8
lrwxrwxrwx 1 root root 7 nov  7 16:19 /dev/mapper/asmdata-vol8 -> ../dm-9
# asm createdisk vol6 /dev/mapper/asmdata-vol6
# asm createdisk vol7 /dev/mapper/asmdata-vol7
# asm createdisk vol8 /dev/mapper/asmdata-vol8
# asm list
mypath/vol3              1.00 GB                  [-] sdd                      
vol1                     4.00 GB                  [-] sdb                      
vol2                     4.00 GB                  [-] sdc                      
vol4                    19.99 GB                  [-] sde1                     
vol6                     1.00 GB       [asmdata-vol6] dm-7                     
vol7                     1.00 GB       [asmdata-vol7] dm-8                     
vol8                     1.00 GB       [asmdata-vol8] dm-9               
# ls -ald /dev/oracleasm/*
brw-rw---- 1 grid asmdba 8, 16 nov  7 16:22 /dev/oracleasm/vol1
brw-rw---- 1 grid asmdba 8, 32 nov  7 16:22 /dev/oracleasm/vol2
brw-rw---- 1 grid asmdba 8, 65 nov  7 16:22 /dev/oracleasm/vol4
lrwxrwxrwx 1 root root       7 nov  7 16:21 /dev/oracleasm/vol6 -> ../dm-7
lrwxrwxrwx 1 root root       7 nov  7 16:21 /dev/oracleasm/vol7 -> ../dm-8
lrwxrwxrwx 1 root root       7 nov  7 16:21 /dev/oracleasm/vol8 -> ../dm-9
# ls -ald /dev/dm-*
brw-rw---- 1 root disk   253, 0 nov  7 16:21 /dev/dm-0
brw-rw---- 1 root disk   253, 1 nov  7 16:21 /dev/dm-1
brw-rw---- 1 root disk   253, 2 nov  7 16:21 /dev/dm-2
brw-rw---- 1 root disk   253, 3 nov  7 16:21 /dev/dm-3
brw-rw---- 1 root disk   253, 4 nov  7 16:21 /dev/dm-4
brw-rw---- 1 root disk   253, 5 nov  7 16:21 /dev/dm-5
brw-rw---- 1 root disk   253, 6 nov  7 16:21 /dev/dm-6
brw-rw---- 1 grid asmdba 253, 7 nov  7 16:21 /dev/dm-7
brw-rw---- 1 grid asmdba 253, 8 nov  7 16:21 /dev/dm-8
brw-rw---- 1 grid asmdba 253, 9 nov  7 16:21 /dev/dm-9


Multipath volumes

If you have device-mapper-multipath installed and configured, you can use multipathed devices. As multipath is managed by device-mapper the way to work with them is the same: just configure /dev/mapper/mpathX as device name similar to using LVM logical volumes. Output then looks like this:

# asm disks
/dev/sda [2:0:0:0] 20.00 GB partitioned
/dev/sdb [2:0:1:0] 1.00 GB multipath (mpatha)
/dev/sde [2:0:4:0] 4.00 GB multipath (mpathe)
/dev/sdf [2:0:5:0] 2.00 GB multipath (mpathf)
/dev/sdg [2:0:6:0] 1.00 GB multipath (mpathg)

EMC Powerpath

EMC Powerpath is an excellent alternative for standard multipathing. Powerpath devices should work.

ScaleIO SCINI devices

DellEMC ScaleIO is a software defined storage layer using a driver named scini. ASMdisks supports SCINI block volumes once they are created using the ScaleIO client (sdc) software.

ASMTab file

We keep track of volume definitions in a file called /etc/asmtab. Let's see what the file looks like:

# /etc/asmtab - configuration file for asmdisks
# Definitions for IORate testing - volumes under /dev/iorate will have root:iops @ mode 0601
PATH=iorate:root:iops:0660
# PATH=oracleasm:oracle:oinstall:0660
#
# This file keeps track of udev disk mappings for asmdisk(1)
# You should normally not have to edit this file directly
# Use asm(1) instead.
#
# On each line:
#
# label type identifier
# where
# label: diskstring/volume name (default diskstring is oracleasm and can be omitted)
# type:  one of scsi, part or mapper (scsi=entire SCSI disk, part=scsi disk partition, mapper=linux disk mapper device)
# label: scsi_id, scsi_id:partition, mapper_name
#
# Ownerships and permissions can be specified for a diskstring:
# PATH=diskstring:owner:group:mode
# default is oracleasm:grid:asmdba:0660
#
# example:
# vol1 scsi 36000c29f825cd85b5fcc70a1aadebf0c   # entire SCSI disk
# vol2 part 36000c298afa5c31b47fe76cbd1750937:1 # partition 1 of entire SCSI disk
# vol3 mapper mpathb                            # /dev/mapper/mpathb (multipath device)
# iorate/test1 mapper iops-vol1                 # LV vol1 on VG iops, will be mapped as /dev/iorate/test1
# -----------------------------------------------
vol2 scsi 36000c297ed46c64ae28ba422880902ca
mypath/vol3 scsi 36000c29df4ec0cef21cfcc31108708a5
vol4 scsi 36000c29997d99a762b3890fc6438cfd1:1
vol6 mapper asmdata-vol6
vol7 mapper asmdata-vol7
vol8 mapper asmdata-vol8
vol1 scsi 36000c2927d96084fab336839b9b7ca68

You see two types of entries in this file: the first is the PATH definition. Default is oracleasm:grid:asmdba:0660 if not specified differently. A definition that comes pre-set in the asmtab file is iorate:iops:0660 which helps us doing raw I/O testing on devices using another excellent EMC tool named <a title="IOrate on Google code" href="https://sites.google.com/site/vwiorate/home" target="_blank">IORate</a>. (using asmdisks this prevents us from running iorate as root and thereby risking overwriting real data).

Updating udev definitions

If you manually changed asmtab you need to re-generate the udev rules. This is done via the "asm scandisks" command. In some cases Linux udev might leave old volume definitions behind, a reboot or manual removal (i.e. rm -f /dev/oracleasm/*) might be required (followed by another run of "asm scandisks" or "udevadm trigger".

Changing default user, group and modes

Some environments like to use different owners and groups than grid:asmdba. If you want to change this, say, to oracle:asmdba, you can enter this in the PATH= definition. The mode may also be changed if you so like (not recommended if you use ASM).

Renaming disks

Sometimes you just don't like a certain name for a volume. FYI, Oracle ASM couldn't care less what the linux name is of the device as long as it is accessible under ASM_DISKSTRING. But maybe you like to assign more useful names to asmdisks volumes for administrative reasons.

# asm list
mypath/vol3              1.00 GB                  [-] sdd
vol1                     4.00 GB                  [-] sdb
# asm renamedisk vol1 data
# asm list
mypath/vol3              1.00 GB                  [-] sdd
data                     4.00 GB                  [-] sdb

Note that I recommend to stop Oracle ASM before renaming disks to avoid ASM errors. Could we move a disk between paths? Sure, let's move mypath/vol3 to oracleasm/vol3:

asm renamedisk mypath/vol3 vol3
# asm list
data                     4.00 GB                  [-] sdb                      
vol2                     4.00 GB                  [-] sdc                      
vol3                     1.00 GB                  [-] sdd                      

Database cloning

If you're familiar with VMware and/or EMC technology you are probably very familiar with the concept of database cloning. You can use these technologies to quickly create a cloned set of volumes, mount these on another host and bring it up as another database, use it for backups, or whatever you like.

In VMware this means that asm disks will get different scsi_id signatures. As "asmdisks" uses these signatures for persistent naming, a cloned set of disks will not be recognised. If the original devices were used by Oracle ASM, the volumes themselves will have ASM signatures written to them so in the case of ASM, we can import the volumes using the "asm import" function. The volumes will get the standard names vol01, vol02, ... etc because ASM does not care what the devs are named anyway and you're free to rename them yourself afterwards.

As illustration I removed all entries from asmtab and rebooted my host:

# asm disks
/dev/sda    [2:0:0:0]  20.00 GB partitioned 
/dev/sdb    [2:0:1:0]   4.00 GB ASM volume 
/dev/sdc    [2:0:2:0]   4.00 GB ASM volume 
/dev/sdd    [2:0:3:0]   1.00 GB ASM volume 
/dev/sde    [2:0:4:0]  20.00 GB partitioned 
/dev/sdf    [2:0:8:0]   2.00 GB ASM volume 
/dev/sdg    [2:0:9:0]   2.00 GB LVM Volume 
/dev/sdh    [3:0:8:0]   2.00 GB LVM Volume 
/dev/sdi    [3:0:9:0]   2.00 GB available 
/dev/sdj    [4:0:8:0]   2.00 GB available 
/dev/sdk    [4:0:9:0]   2.00 GB available 
/dev/sdl    [5:0:8:0]   2.00 GB available 
/dev/sdm    [5:0:9:0]   2.00 GB available 
/dev/sdn   [5:0:14:0]  16.00 GB available

You see asmdisks recognizes ASM signatures but does not know where to configure the volumes so on the "cloned" host ASM cannot see those volumes. Let's import the volumes:

# asm import 
importing /dev/sdb as /dev/oracleasm/vol01
importing /dev/sdc as /dev/oracleasm/vol02
importing /dev/sdd as /dev/oracleasm/vol03
importing /dev/sdf as /dev/oracleasm/vol04
# asm list
vol01                    4.00 GB                  [-] sdb                      
vol02                    4.00 GB                  [-] sdc                      
vol03                    1.00 GB                  [-] sdd                      
vol04                    2.00 GB                  [-] sdf

Starting a SQLPlus session as sysasm:

SYS:+ASM > @ls-asmdisks

DISKGROUP    DISKNAME        TOTAL_MB USED_MB    FREE_MB MOUNTED STATUS       STATE    PATH
------------ -------------- -------- ------- ---------- ------- ---------- -------- ------------------------------
                   0       0          0 CLOSED    MEMBER       NORMAL   /dev/oracleasm/vol01
                   0       0          0 CLOSED    MEMBER       NORMAL   /dev/oracleasm/vol02
                   0       0          0 CLOSED    MEMBER       NORMAL   /dev/oracleasm/vol03
                   0       0          0 CLOSED    FORMER       NORMAL   /dev/oracleasm/vol04
SYS:+ASM > @ls-diskgroups

    GRPNUM NAME       BLKSZ AU_SIZE STATE    TYPE   TOTAL_MB FREE_MB USABLE_MB    OFFL
---------- ---------- ----- ------- ----------- ------ -------- ------- --------- ----------
     0 REDO        4096      0 DISMOUNTED              0       0     0       0
     0 DATA        4096      0 DISMOUNTED              0       0     0       0
     0 TEMP        4096      0 DISMOUNTED              0       0     0       0

SYS:+ASM > alter diskgroup redo mount;

Diskgroup altered.

SYS:+ASM > alter diskgroup data mount;

Diskgroup altered.

SYS:+ASM > alter diskgroup temp mount;

Diskgroup altered.
SYS:+ASM > @ls-diskgroups

    GRPNUM NAME       BLKSZ AU_SIZE STATE    TYPE   TOTAL_MB FREE_MB USABLE_MB    OFFL
---------- ---------- ----- ------- ----------- ------ -------- ------- --------- ----------
     1 REDO        4096      1 MOUNTED    EXTERN       1024     335       335       0
     2 DATA        4096      8 MOUNTED    EXTERN       4096    3128      3128       0
     3 TEMP        4096      8 MOUNTED    EXTERN       4096    3912      3912       0

Note that a restart of ASM would usually also mount all diskgroups (preventing you to do this manually).

ASMstat

Ever used "iostat" under Linux and had issues figuring out which Linux device was mapped to what ASM device?

Because we know what's mapped here we can do some translation. "asmstat" is a small wrapper for "iostat" that filters out some irrelevant disks (like cdrom) and translates /dev/sdXX names into /dev/oracleasm/XXX names. All parameters that you give "asmstat" are forwarded to "iostat" without further modification.

Example:

# iostat 
Linux 2.6.32-504.el6.x86_64 (db01)     07-11-14     _x86_64_    (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3,30    0,00    2,67    1,05    0,00   92,98

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               6,13       303,14        15,49     460918      23546
sdb               1,44         9,82         2,30      14931       3496
sdc               1,92        13,76         2,28      20928       3472
[snip]
dm-7              1,37         1,78         1,70       2710       2580
dm-8              0,40         1,40         0,09       2124        140
dm-9              4,23       221,55        13,22     336874      20096
# asmstat -xm 2
Linux 2.6.32-504.el6.x86_64 (db01)     07-11-14     _x86_64_    (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3,28    0,00    2,65    1,01    0,00   93,06

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sda                  2,40     1,42    4,62    1,26     0,14     0,01    51,76     0,01    1,38   0,73   0,43
vol01                0,07     0,00    1,11    0,29     0,00     0,00     8,39     0,01    5,04   5,01   0,70
vol02                0,08     0,00    1,56    0,29     0,01     0,00     8,35     0,01    4,05   4,03   0,75
vol03                0,06     0,00    0,91    0,29     0,00     0,00     8,42     0,01    5,59   5,57   0,67
sde                  0,05     0,00    0,99    0,00     0,00     0,00     8,36     0,01   14,06  13,62   1,35
vol04                0,10     0,00    1,33    0,00     0,01     0,00     8,58     0,00    1,47   1,44   0,19
sdg                  0,01     0,00    0,78    0,00     0,00     0,00     8,06     0,00    2,66   1,70   0,13
sdi                  0,05     0,00    0,58    0,00     0,00     0,00     8,74     0,00    2,00   1,95   0,11
sdh                  0,01     0,00    0,48    0,00     0,00     0,00     8,11     0,00    3,26   2,58   0,12
sdl                  0,07     0,00    0,58    0,00     0,00     0,00     8,98     0,00    1,95   1,91   0,11
sdj                  0,05     0,00    0,49    0,00     0,00     0,00     8,77     0,00    2,63   2,57   0,13
sdm                  0,05     0,00    0,58    0,00     0,00     0,00     8,75     0,00    1,86   1,80   0,10
sdn                  0,05     0,00    0,59    0,00     0,00     0,00     8,73     0,00    0,53   0,53   0,03
sdk                  0,04     0,00    0,50    0,00     0,00     0,00     8,70     0,00    2,16   2,16   0,11

Wipedisk

Wipedisk is another small script in the asmdisks package. It backs up the first 1MiB of a block device to /tmp/ and then zeroes it out. Very handy to clear old ASM disks or even a boot disk for re-install of the OS. Because this is potentially a dangerous command (even with the data backed up before clearing) it requires the "-f" flag to become effective and there's a 5 second countdown in which you may still abort.

# wipedisk -f /dev/sdn
Backing up 1st MB of /dev/sdn in /tmp/_dev_sdn, restore with dd if=/tmp/_dev_sdn of=/dev//dev/sdn bs=1M count=1
1+0 records in
1+0 records out
1048576 bytes (1,0 MB) copied, 0,0128343 s, 81,7 MB/s
Wiping 1st mb of disk /dev/sdn
5
4
3
2
1
1+0 records in
1+0 records out
1048576 bytes (1,0 MB) copied, 0,0854113 s, 12,3 MB/s
Read the wipedisk(1) man page for more info.
  1. Requires the "bash-completion" package to be installed