If you’ve read our previous articles on LIO , you’ve probably gathered that LIO is one of our favorite Linux utilities. We love the ability to use inexpensive hardware and FC or iSCSI cards to create a rock-solid Linux-based SAN to provide back-end storage for Hyper-V cluster shared volumes, highly-available shared VHDXs, or LUNs for Windows File Servers. We also love the flexibility that Linux MDADM/LVM offers to seamlessly add or expand storage arrays or add new LUNs. It really gives the IT Pro the ability to use many Enterprise features in a home lab you’d only be able to otherwise replicate with expensive, impractical hardware.
In the end, all this flexibility means we will inevitably tinker with configurations, add and remove hardware, and just generally screw around with things until we break them, then fix them, then break them again. That’s what we do. And, as it so often goes in IT, with any luck we’ll learn a thing or two along then way.
This was exactly the case when we recently expanded one of our backup Ubuntu SANs by adding a new disk. After the new volume was added, it became apparent that the previous method of using the typical Linux device notation for harddisks (/dev/sda, /dev/sdb, etc.) was not an optimal configuration.
Consider the following LIO backstores configuration:
1 2 3 4 5 6 7 8 9 |
/backstores/iblock> ls o- iblock ………………………………………………………………………………………… [5 Storage Objects] o- fc_lun00_backupdisk00 ………………………………………………………………………… [/dev/sda activated] o- fc_lun01_backupdisk01 ………………………………………………………………………… [/dev/sdb activated] o- fc_lun02_backupdisk02 ………………………………………………………………………… [/dev/sdc activated] o- fc_lun03_backupdisk03 ………………………………………………………………………… [/dev/sdd activated] o- fc_lun04_backupdisk04 ………………………………………………………………………… [/dev/sdg activated] |
This configuration has worked fine for months. However, after adding new disk, we quickly realized one of the volumes being presented to a two-node Hyper-V cluster was now listed as “Offline – Not initialized,” and any attempts to bring it online failed with I/O errors.
Looking at the backup uSAN, the disk that was formerly /dev/sdg was now /dev/sdh, and LIO’s ACLs were no longer correct. Though quick and dirty, using the /dev/sdx notation is clearly not the best way to add a single disk to the LIO backstores, since these values are subject to change. Looking in /dev/disk, we see a few different options that may be helpful:
1 2 3 4 5 6 7 8 9 10 11 12 |
$ ll /dev/disk total 0 drwxr-xr-x 8 root root 160 May 22 14:20 ./ drwxr-xr-x 16 root root 4620 Sep 23 21:40 ../ drwxr-xr-x 2 root root 920 Sep 23 21:40 by-id/ drwxr-xr-x 2 root root 160 Sep 23 21:40 by-label/ drwxr-xr-x 2 root root 100 Sep 23 21:40 by-partlabel/ drwxr-xr-x 2 root root 180 Sep 23 21:40 by-partuuid/ drwxr-xr-x 2 root root 200 Sep 23 20:30 by-path/ drwxr-xr-x 2 root root 180 Sep 23 21:40 by-uuid/ |
Typically by-UUID is a good option – we’ve used it in the past for other operations. However, we’re specifically exporting block devices, and UUID only shows disks with partitions. /dev/disk/by-label, /dev/disk/by-partlabel, and /dev/disk/by-partuuid are much the same way, not all disks will have a label or partitions to view. /dev/disk/by-path is promising, but only if all the relevant disks are hanging off a SAS controller. Since many lab environments, such as ours, may make use of both on-board SATA headers as well as PCIe SAS controllers, that only leaves /dev/disk/by-id. Now listing disks by /dev/disk/by-id appears a bit messy at first, but if you look carefully you’ll a see neat and tidy way of referencing disks.
Specifically, let’s look a this systems’ disk in question, /dev/sdh.
1 2 3 4 |
$ ll /dev/disk/by-id | grep sdh lrwxrwxrwx 1 root root 9 Sep 25 23:55 ata-SAMSUNG_HD204UI_S2H7J1SZ914444 -> ../../sdh |
Very nice — we see a disk ID that not only tells us the make and model of the drive, but also appends the drive’s serial number to the end. This is quite handy in a system with 10 or 20 drives, many of which may be the same model. Now, let’s go back to LIO’s targetcli and try to add the block device using this new identification, rather than the device letter.
1 2 3 4 5 6 7 8 9 10 11 12 13 |
/backstores/iblock> create name=fc_lun04_backupdisk04 dev=/dev/disk/by-id/ata-SAMSUNG_HD204UI_S2H7J1SZ914444 Generating a wwn serial. Created iblock storage object fc_lun04_backupdisk04 using /dev/disk/by-id/ata-SAMSUNG_HD204UI_S2H7J1SZ914444. /backstores/iblock> ls o- iblock ………………………………………………………………………………………… [5 Storage Objects] o- fc_lun00_backupdisk00 ………………………………………………………………………… [/dev/sda activated] o- fc_lun01_backupdisk01 ………………………………………………………………………… [/dev/sdb activated] o- fc_lun02_backupdisk02 ………………………………………………………………………… [/dev/sdc activated] o- fc_lun03_backupdisk03 ………………………………………………………………………… [/dev/sdd activated] o- fc_lun04_backupdisk04 …………………………………. [/dev/disk/by-id/ata-SAMSUNG_HD204UI_S2H7J1SZ914444 deactivated] |
OK. Looks like that works just fine. Now, let’s create the associated LUN.
1 2 3 4 5 6 7 8 9 10 11 12 |
/qla2xxx/21:0…88:62:a6/luns> create lun=4 storage_object=/backstores/iblock/fc_lun04_backupdisk04 Successfully created LUN 4. Created mapped LUN 4 in node ACL 21:00:00:1b:32:86:d2:45 /qla2xxx/21:0…88:62:a6/luns> ls o- luns ……………………………………………………………………………………………………. [5 LUNs] o- lun0 …………………………………………………………………….. [iblock/fc_lun00_backupdisk00 (/dev/sda)] o- lun1 …………………………………………………………………….. [iblock/fc_lun01_backupdisk01 (/dev/sdb)] o- lun2 …………………………………………………………………….. [iblock/fc_lun02_backupdisk02 (/dev/sdc)] o- lun3 …………………………………………………………………….. [iblock/fc_lun03_backupdisk03 (/dev/sdd)] o- lun4 ……………………………….. [iblock/fc_lun04_backupdisk04 (/dev/disk/by-id/ata-SAMSUNG_HD204UI_S2H7J1SZ914444)] |
Again, all appears well. To summarize the final steps, we then added the storage in Failover Cluster Manager, created the Cluster Shared Volume from the new disk, created a VHDX to fill the LUN, and attached the new virtual disk to our virtual machine with out issue. Now, when we add or swap drives, change disk controllers, or even completely move disks to new motherboard/chassis, we no longer have to worry about device letters, as this new (and better) method removes any ambiguity as to which disk is which.