Without a doubt, modern virtualization has changed the landscape of enterprise computing forever. Since virtual machines are abstracted away from the physical hardware, changes in compute, memory, and storage resources become mere clicks of a mouse. And, as hypervisors mature, many operations that were once thought of as out-of-band tasks, such as adding storage or even memory can now be done with little, or even zero downtime.
Hyper-V SCSI Disks and Linux
In many cases, hypervisors are backed by large storage area networks (SANs). This provides shared storage for hypervisor nodes that supports failover clustering and high availability. Additionally, it gives administrators the ability to scale the virtual environment, including the ability to easily add or expand storage on existing virtual servers. Microsoft’s Hyper-V 2012 introduced Generation 2 VMs, which extends this functionality. Among the many benefits of Gen2 VMs, was the ability to boot from a SCSI disk rather than IDE. This requires UEFI rather than a legacy BIOS, so it’s only supported among newer operating systems. Many admins I talk to think this is limited to Microsoft Server 2012 and newer, probably because of the sub-optimal phrasing in the Hyper-V VM creation UI that altogether fails to mention Linux operating systems.
The fact is, however, that many newer Linux OSes also support this ability, as shown in these tables from Microsoft.
More Disk, Please
Once you’ve built a modern Linux VM and you’re booting from synthetic SCSI disks rather than emulated IDE drives, you gain numerous advantages, not the least of which is the ability to resize the OS virtual hard disk (VHDX) on the fly. This is really handy functionality – after all, what sysadmin hasn’t had an OS drive run low on disk space at some point in their career? This is simply done from the virtual machine settings in Hyper-V Manager or Failover Cluster Manager by editing the VHDX.
Now, if you’re a Microsoft gal or guy, you already know that what comes next is pretty straightforward. Open the Disk Management MMC, rescan the disks, extend the file system, and viola, you now automagically have a bigger C:\ drive. But what about for Linux VMs? Though it might be a little less intuitive, we can still accomplish the same goal of expanding the primary OS disk with zero down time in Linux.
On-the-Fly Resizing
To demonstrate this, let’s start with a vanilla, Hyper-V Generation 2, CentOS 7.6 VM with a 10GB VHDX attached to a SCSI controller in our VM. Let’s also assume we’re using the default LVM partitioning scheme during the CentOS install. Looking at the block devices in Linux, we can see that we have a 10GB disk called sda which has three partitions – sda1, sda2 and sda3. We’re interested in sda3, since that contains our root partition, which is currently 7.8GB, as demonstrated here by the lsblk command.
1 2 3 4 5 6 7 8 9 10 11 |
# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 10G 0 disk ├─sda1 8:1 0 200M 0 part /boot/efi ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 8.8G 0 part ├─centos-root 253:0 0 7.8G 0 lvm / └─centos-swap 253:1 0 1G 0 lvm [SWAP] sr0 11:0 1 1024M 0 rom |
Now let’s take a look at df. Here we can see an XFS filesystem on our 7.8GB partition, /dev/mapper/centos-root which is mounted on root.
1 2 3 4 5 6 7 8 9 10 11 12 |
# df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/centos-root xfs 7.8G 1007M 6.9G 13% / devtmpfs devtmpfs 479M 0 479M 0% /dev tmpfs tmpfs 490M 0 490M 0% /dev/shm tmpfs tmpfs 490M 6.8M 484M 2% /run tmpfs tmpfs 490M 0 490M 0% /sys/fs/cgroup /dev/sda2 xfs 1014M 124M 891M 13% /boot /dev/sda1 vfat 200M 12M 189M 6% /boot/efi tmpfs tmpfs 98M 0 98M 0% /run/user/0 |
Finally, let’s have a look at our LVM summary:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
# pvs PV VG Fmt Attr PSize PFree /dev/sda3 centos lvm2 a-- 8.80g 0 # vgs VG #PV #LV #SN Attr VSize VFree centos 1 2 0 wz--n- 8.80g 0 # lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert root centos -wi-ao---- 7.80g swap centos -wi-ao---- 1.00g |
From this information we can see that there’s currently no room to expand our physical volume or logical volume, as the entirety of /dev/sda is consumed. In the past, with a Gen1 Hyper-V virtual machine, we would have had to shut the VM down and edit the disk, since it used an emulated IDE controller. Now that we have a Gen2 CentOS VM with a SCSI controller, however, we can simply edit the disk on the fly, expanding it to 20GB.
Once the correct virtual disk is located, select the “Expand” option.
Next, provide the size of the new disk. We’ll bump this one to 20GB.
Finally, click “Finish” to resize the disk. This process should be instant for dynamic virtual hard disks, but may take a few seconds to a several minutes for fixed virtual hard disks, depending on the size of the expansion and speed of your storage subsystem. You can then verify the new disk size by inspecting the disk.
OK, so we’ve expanded the VHDX in Hyper-V, but we haven’t done anything to make our VM’s operating system aware of the new space. As seen here with lsblk, the OS is indifferent to the expanded drive.
1 2 3 4 5 6 7 8 9 10 11 |
# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 10G 0 disk ├─sda1 8:1 0 200M 0 part /boot/efi ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 8.8G 0 part ├─centos-root 253:0 0 7.8G 0 lvm / └─centos-swap 253:1 0 1G 0 lvm [SWAP] sr0 11:0 1 1024M 0 rom |
Taking a look at parted, we again see that our /dev/sda disk is still showing 10.7GB. We need to make the CentOS operating system aware of the new space. A reboot would certainly do this, but we want to perform this entire operation with no downtime.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# parted GNU Parted 3.1 Using /dev/sda Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) p Model: Msft Virtual Disk (scsi) Disk /dev/sda: 10.7GB Sector size (logical/physical): 512B/4096B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 1049kB 211MB 210MB fat16 EFI System Partition boot 2 211MB 1285MB 1074MB xfs 3 1285MB 10.7GB 9452MB lvm |
Issue the following command to rescan the relevant disk – sda in our case. This tells the system to rescan the SCSI bus for changes, and will report the new space to the kernel without a restart.
1 2 3 |
# echo 1 > /sys/block/sda/device/rescan |
Now, when we look at parted again, we’re prompted to move the GPT table to the back of the disk, since the secondary table is no longer in the proper location after the VHDX expansion. Type “Fix” to correct this, and then once again to edit the GPT to use all the available disk space. Once this is complete, we can see that /dev/sda is now recognized as 20GB, but our sda3 partition is still only 10GB.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
# parted GNU Parted 3.1 Using /dev/sda Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) p Error: The backup GPT table is not at the end of the disk, as it should be. This might mean that another operating system believes the disk is smaller. Fix, by moving the backup to the end (and removing the old backup)? Fix/Ignore/Cancel? Fix Warning: Not all of the space available to /dev/sda appears to be used, you can fix the GPT to use all of the space (an extra 20971520 blocks) or continue with the current setting? Fix/Ignore? Fix Model: Msft Virtual Disk (scsi) Disk /dev/sda: 21.5GB Sector size (logical/physical): 512B/4096B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 1049kB 211MB 210MB fat16 EFI System Partition boot 2 211MB 1285MB 1074MB xfs 3 1285MB 10.7GB 9453MB lvm |
Next, from the parted CLI, next use the resizepart command to grow the partition to the end of the disk.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
(parted) resizepart Partition number? 3 End? [10.7GB]? 21.5GB (parted) p Model: Msft Virtual Disk (scsi) Disk /dev/sda: 21.5GB Sector size (logical/physical): 512B/4096B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 1049kB 211MB 210MB fat16 EFI System Partition boot 2 211MB 1285MB 1074MB xfs 3 1285MB 21.5GB 20.2GB lvm |
Our sda3 partition is now using the maximum space available, 20.2GB. The lsblk command also now correctly reports our disk as 20GB.
1 2 3 4 5 6 7 8 9 10 11 |
# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 20G 0 disk ├─sda1 8:1 0 200M 0 part /boot/efi ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 18.8G 0 part ├─centos-root 253:0 0 7.8G 0 lvm / └─centos-swap 253:1 0 1G 0 lvm [SWAP] sr0 11:0 1 1024M 0 rom |
But what about our LVM volumes? As suspected, our physical volumes, volume groups and logical volumes all remain unchanged.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
# pvs PV VG Fmt Attr PSize PFree /dev/sda3 centos lvm2 a-- 8.80g 0 # vgs VG #PV #LV #SN Attr VSize VFree centos 1 2 0 wz--n- 8.80g 0 # lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert root centos -wi-ao---- 7.80g swap centos -wi-ao---- 1.00g |
We need to first tell our pv to expand into the available disk space on the partition. Do this with the pvresize command as follows:
1 2 3 4 5 6 7 8 9 |
# pvresize /dev/sda3 Physical volume "/dev/sda3" changed 1 physical volume(s) resized or updated / 0 physical volume(s) not resized # pvs PV VG Fmt Attr PSize PFree /dev/sda3 centos lvm2 a-- 18.80g 10.00g |
Sure enough, our pv is now 18.8GB with 10.00GB free. Now we need to extend the logical volume and it’s associated filesystem into the free pv space. We can do this with a single command:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
# lvextend -l +100%FREE -r /dev/mapper/centos-root Size of logical volume centos/root changed from 7.80 GiB (1997 extents) to 17.80 GiB (4557 extents). Logical volume centos/root successfully resized. meta-data=/dev/mapper/centos-root isize=512 agcount=4, agsize=511232 blks = sectsz=4096 attr=2, projid32bit=1 = crc=1 finobt=0 spinodes=0 data = bsize=4096 blocks=2044928, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal bsize=4096 blocks=2560, version=2 = sectsz=4096 sunit=1 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 data blocks changed from 2044928 to 4666368 |
Looking at our logical volumes confirms that our root lv is now 17.80GB of the 18.80GB total, or exactly 10.0GB larger than we started with, as one would expect to see.
1 2 3 4 5 6 |
# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert root centos -wi-ao---- 17.80g swap centos -wi-ao---- 1.00g |
A final confirmation with the df command illustrates that our XFS root filesystem was also resized.
1 2 3 4 5 6 7 8 9 10 11 12 |
# df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/centos-root xfs 18G 1008M 17G 6% / devtmpfs devtmpfs 479M 0 479M 0% /dev tmpfs tmpfs 490M 0 490M 0% /dev/shm tmpfs tmpfs 490M 6.8M 484M 2% /run tmpfs tmpfs 490M 0 490M 0% /sys/fs/cgroup /dev/sda2 xfs 1014M 124M 891M 13% /boot /dev/sda1 vfat 200M 12M 189M 6% /boot/efi tmpfs tmpfs 98M 0 98M 0% /run/user/0 |
Conclusion
So there you have it. Despite some hearsay to the contrary, modern Linux OSes run just fine as Gen2 VMs on Hyper-V. Coupled with a SCSI disk controller for the OS VHDX, this yields the advantage of zero-downtime root partition resizing in Linux, though it’s admittedly a few more steps than a Windows server requires. And though Linux on Hyper-V might not seem like the most intuitive choice to some sysadmins, Hyper-V has matured significantly over the past several releases and is quite a powerful and stable platform for both Linux and Windows. And one last thing – when you run critically low on disk space on Linux, don’t forget to check those reserved blocks for a quick fix!