Samsung PM853T SSD Review

Samsung PM853T SSD Review

Here at Teknophiles, we don’t believe in a once-size-fits-all approach to selecting hard drives for our lab servers. We prefer to adhere to the rule of specificity, where drives have a defined purpose and drive selection is based on several criteria that suit that purpose. In no particular order, we evaluate capacity, cost, reliability, performance and form factor when selecting a drive for a particular role.

Looking at this list of attributes, it’s easy to reach the conclusion that simply selecting the fastest drive would be a no-brainer for all applications. But fast drives come at an expense – both literal expense, as well as capacity expense. And, frankly, there are times where you just don’t need the the capacity or even the raw performance that some drives offer. One example, as detailed in our Silicon Power S60 60GB SSD Review, are server OS drives. On nearly every server we build, any serious workload is going to be performed on a dedicated array or SAN LUN, where IOPS and throughput are known quantities that are appropriately sized. As such, dedicated operating system drives typically experience low I/O and are approximately 75-80% read operation. You just won’t see much benefit by spending extra cash on a blazing fast SSD for your OS. And when you further consider that we nearly always configure our OS drives in RAID-1, even relatively “slow” SSDs will yield perfectly usable read speeds for a typical operating system. Heck, reliability even takes somewhat of a back seat when using cheaper drives in a RAID-1 configuration – by simply by keeping an extra $30 drive or two on hand as spares, you’ll still come out ahead financially, with no little down time waiting on a new drive to arrive.

Have Your Cake and Eat It Too

But what about those times where you do need speed, capacity, and reliability? It’s sorta like that old muscle car adage: cheap, fast, reliable – pick any two. The same premise generally holds true for computer components, including hard drives. Simply put, if you want fast and cheap, it likely won’t be reliable. Reliable and cheap? It’s not gonna be fast. You want all three? Unfortunately, you’re going to have to pay for it.

Or are you?

Perhaps there’s a happy medium – as long as you understand what it is you’re looking for. You see, performance is relative. There are certainly applications that require an abundance of IOPS. Others require significant write endurance. Others yet are heavily read biased. For each of these use cases, there are drives that fit the specific profile. For us, we needed some reliable, reasonably fast drives that will be 80% read-biased, but not break the bank. We’ve think we’ve found the sweet spot with the Samsung PM853T.

The Samsung PM853T

The Samsung PM853T series drives were mass produced around 2014-2016, so all the drives floating around out there are data center pulls, some with low hours, and in some cases, even New Old-Stock (NOS). Still, these are a great deal and can be had for as low as $0.10 per GB. Keep in mind that many of these drives are OEM drives that were sold bundled with servers, and thus will carry no warranty from the drive manufacturer even if they had a recent manufacture date. At this price point, however, having a cold-spare on hand is certainly achievable, and is highly recommended.

Samsung considers these drives to be a mixed workload drive with high sustained performance, which is perfect for our purposes. Note that the PM853T is an TLC SATA III 6 Gb/s drive, so like other SATA SSDs, it’s limited to a theoretical 600 MB/s. In our case, this is mostly irrelevant, however, as we’ll be using these in RAID-1/0 arrays as the disk subsystem for Hyper-V clusters. Given a minimum of 4 disks in an array (and possibly many more), this configuration can easily saturate the 2000 MB/s maximum throughput of a single 4-lane SFF-808x connector on an older SAS2 HBA like the LSI-Avago SAS 9210-8i.

Attributes

Samsung offered the PM853T in 240 / 480 / 960 GB sizes, and the drive offers many features not found on Samsung’s consumer drives.

Samsung PM853T – Specifications
Form factor 2.5 inches
Capacity 240 / 480 / 960 GB
Host Interface SATA3 – 6 Gb/s
Encryption AES 256-bit Hardware Encryption
Mean time 2.0 million hours
Uncorrectable bit 1 in 10^17
Power consumption Active Read/Write : 2.7 Watt/3.8 Watt, Idle : 1.2 Watt
TBW – 240 GB : 150 TBW
– 480 GB : 300 TBW
– 960 GB : 600 TBW
Cache power protection Supported
Sequential R/W (MB/s) Up to 530 / 420 MB/s
Random R/W (IOPs) Up to 90,000 / 14,000
Physical dimensions 100mm x 70mm x 7mm
Weight 63g

Among the features, Samsung lists the following:

    Consistent high-quality performance. Delivers consistent performance under diverse workloads to meet various data center demands.
    Advanced Error-Correcting Code (ECC) engine. Corrects read failures to greatly improve the reliability of the data stored in the memory for higher error correction and endurance than the BCH code can deliver alone.
    End-to-end protection. Extends error detection to cover the entire path, from the host interface to the NAND flash memory in the SSD for superior data integrity.
    Power-loss protection. Ensures no data loss during unexpected power failures by using the power supply of tantalum capacitors to borrow enough time to store all cached data to flash memory.
    SMART technology. Anticipates failures and warns users of impending drive failure, enabling time to replace the ailing drive to avoid data loss and system failure malfunctions.
    Thermal throttling. Regulates the temperature of the hardware components automatically to protect them from overheating by managing its performance level to prevent data loss.

Performance

So how does it perform? Samsung provides an enormous amount of data in their product brief on the PM853T, but here are some highlights of tests conducted in Samsung’s data lab using a PM853T 480 GB drive against a competitor’s product. Samsung uses the following tools to generate this data: Fio 2.1.3, Jetstress, and IOMeter.

Sustained Performance Tests

In this test, Samsung pitted the PM853T against an competitor’s MLC SSD drive during an 11 hour workload. The results indicate that the Samsung drive shows much lower latency with less standard deviation (more consistency). Overall the Samsung drive also had overall higher average IOPS.

Read/Write Tests

Additionally, the Samsung drive outperformed it’s competitor in both sustained random, as well as sequential read/write tests, achieving nearly 160000 IOPS at 100% random read in RAID-5 configurations, and 30000 IOPS at 100% random write in RAID-1 configurations.

In the sequential read tests throughput reached approximately 1500 MB/s in RAID-1 and over 1200 MB/s write in RAID-5 and outperforms its competitor as much as 29%, depending on RAID configuration and queue depth.

In mixed workloads it’s a similar story – the PM853T performs outperforms its competitor at all queue depths, in both non-RAID and RAID configurations, achieving more than 60,000 IOPs in RAID-1 at a RW ratio of 75:25, which is similar to typical virtual environment workloads.

Latency

In terms of average and maximum latency, the PM853T again performs admirably against a competitor.

Application Workloads

Finally, in both virtual environments using multiple VMs, as well in as various real-world application workloads, the PM853T again outperforms its competitor across the board.

In our own much simpler tests, we used a Samsung PM853T 960 GB drive. This drive was a server pull that, as you can see, had very low hours.

We saw read/write performance very much in line with Samsung’s official claims and consistently saw over sequential reads over 550 MB/s read and sequential writes over 420 MB/s.

Limitations

All this said, these drives do have certain limitations that should be at least touched upon:

    Form Factor. These drives are 2.5″, so they may not fit in your existing NAS, at least not without an adapter. Though at 7 mm z-height these will easily fit in all 2.5″ drive locations.
    SATA III. The PM853T is SATA III, not SAS or PCI-E, so if you need the raw performance of PCI-E or the expanded feature set of SAS such as multiple initiators, full duplex speeds or multipath I/O, then these drives are not for you.
    Write Speed. Being a read-biased drive, one would expect write performance to take a bit of a hit. These drives certainly do not display write speeds as fast as modern PCI-E/NVMe based drives. That said, they’re no slouch either, especially in RAID arrays. And at $0.10 per GB, you can actually afford to build an array with them.
    Endurance. Again, being a read-centric TLC SSD, the PM853T is only rated at 0.3 drive writes per day (DWPD). SLC drives can typically handle as many as 10x the number of write cycles that MLC or TLC drives can. This translates to nearly 300 GB in drive writes daily for 5 years. Unless you have some atypical use case, these drives should last a very long time in a typical 80%/20% R/W virtualization scenario.

Conclusion

So how exactly are we using these drives at Teknophiles? We’re currently running nearly 30 virtual machines on a single 1.8TB RAID-1/0 volume (4 x 960GB Samsung PM853T) and these drives don’t break a sweat. Even when hammering the environment with Windows Updates, mass live migrations or boot storms, these drives hold up well. The PM853T’s random IO performance and low latency makes it quite suitable to meet the demands of the mixed workloads that virtual machines place on the disk subsystem. Additionally, with numerous 853T drives currently in play (4 x 960 GB and 2 x 480 GB), we’ve not had a single failure in more than 10k hours use – these seem to be quite reliable drives. Simply put, for a home Hyper-V or ESX lab, it’s hard to imagine a better drive for the money. Factor in the quite excellent IO and throughput per watt of power consumption these drives produce, and you have a clear winner with the PM853T.

Installing Zabbix 4.0 LTS on CentOS 7

Installing Zabbix 4.0 LTS on CentOS 7

When it comes to enterprise monitoring solutions, there are a myriad of options to choose from, both paid and free and open-source software (FOSS). Here at Teknophiles, we’ve used just about all of them, and we can tell you that many are good, some are even great, but none are perfect. Even at some of the higher price points, we usually find something we don’t care for. Generally speaking, paying good money for monitoring software will get you support, and perhaps some ease of configuration that open source solutions do not offer.

That said, there are numerous free and open-source monitoring options that get the job done quite well. After using Nagios for many years, we’ve recently become quite enamored with Zabbix. Though not trivial to configure, especially in large environments (what monitoring solution is?), Zabbix is stable, polished and quite extensible. In this article – the first of several in a series on installing and configuring Zabbix – we’ll walk through a typical Zabbix install on Linux, using the latest long-term release from Zabbix, so you can take it for a test drive.

Zabbix Server Sizing

To get started, we’ll first assume you have a running CentOS 7 box, either a virtual or physical machine. With respect to hardware requirements, lab/test/home environments shouldn’t require more than 1-2 CPU cores and 2-4 GB of RAM. We’re currently monitoring over 3600 items, with over 2000 triggers on a virtual server with 1 CPU core and 2GB of RAM, and the server rarely shows any significant resource utilization.

Though disk usage depends on many factors, the Zabbix database will be the primary offender when it comes to disk consumption. Once Zabbix is installed and configured, the application files will change very little in terms of size, so whether or not the database is local or remote will greatly impact disk sizing for the Zabbix server. Regardless of the database location, make sure you have enough space for the database to grow as you accumulate historical data. This will be influenced by not only the number of hosts and items monitored, but also historical and trend storage periods. As a general starting point for a lab server with a local database, a root partition of 15-20 GB should suffice. You can see here that with just over 3600 items, our lab database is chewing up approximately 600 MB/month. This will likely stabilize at some point after all data retention periods are reached, but should be a consideration when sizing the disks.

Do some due diligence and size your Zabbix environment appropriately up front, however. Though this can depend on many factors, such as whether your database is local or remote, whether you choose to use MySQL InnoDB vs. PostgreSQL, the number of hosts you wish to monitor and whether or not you choose to use Zabbix Proxies, it’s much easier to provision resources earlier in the process than later, especially on physical hardware. Also, keep in mind that in medium to large installations (500-1000 hosts), disk I/O will also start become an important factor, along with CPU and memory considerations. Plan the disk subsystem accordingly. Some general sizing recommendations from the folks at Zabbix can be found here.

SELinux

For the purposes of this article, we’re also going to assume you’re a responsible Linux sysadmin and will be installing Zabbix with SELinux in enforcing mode. We strongly recommend that you leave SELinux this way. We’re aware that SELinux can be challenging and the knee-jerk reaction by many is to disable SELinux – we’ve been tempted ourselves at times. SELinux naturally adds a few steps to the process, but it’s completely manageable with the tools available, and will ultimately leave you with a better security stance. You can check the current status of SELinux as follows:

Prerequisites

The Zabbix server installation requires several prerequisite components to function – an Apache web server, PHP, and a database server. First, we’ll install the standard Apache httpd server with mod_ssl.

Now start the Apache server and ensure the service persists through reboots.

Next, we need to install PHP along with the necessary PHP modules. First, install yum-utils and enable the Remi PHP repo. In this instance, we’re opting to use PHP 7.1.

Lastly, we’ll install the database. We’re using MariaDB, a open-source and actively-developed fork of MySQL. Install MariaDB as follows:

Now start the MariaDB server and ensure the service persists through reboots.

Next, complete the secure installation for MariaDB.

Finally, create the Zabbix Database with the following mysql commands:

Installing Zabbix

Now that the prerequisites are satisfied, we can proceed with installing the Zabbix application from packages. First, add the Zabbix repo. Make sure you have the URL for the correct version – we want Zabbix 4.0, as shown below.

Now install the Zabbix server and web frontend.

Next, import the initial Zabbix schema into the database using the zabbix database user and password previously created.

NOTE

The ‘zabbix’ parameter after the -p is NOT the password. This is a common misconception – however, the password would have no space after the -p option. In this case, the ‘zabbix’ parameter is specifying the database for the mysql connection to use. You will be prompted to provide the password for the zabbix database user after you enter the command.

Configure the server to connect to the database as shown here. Some of these parameters may already be set correctly in the config, while others may be commented out by default.

Finally, modify the Apache config for Zabbix as follows. Comment out the section for mod_php5.c, replacing it with a stanza for PHP 7, using the parameters below. Restart Apache after saving the config.

Starting the Zabbix Server

We’re now finally ready to start the Zabbix server for the first time.

After attempting to start the server, however, we can see that the service failed to start.



SELinux Configuration

So what’s going on here? We suspect SELinux is interfering with something out-of-the-box, so let’s do some investigation to confirm. First, install the SELinux troubleshooting tools.

Now run the setroubleshoot cli tool to search the audit log for SELinux alerts.

So we can see here just how useful the SELinux troubleshooting tools are. Not only does the utility tell us exactly what is being blocked (Zabbix Server attempting create access on the zabbix_server_alerter.sock sock_file), but it also gives us the exact command we need to resolve the issue. Not so bad, eh? Simply execute the suggested commands to allow the proper access to the zabbix_server_alerter.sock file, as shown here:

Now let’s attempt to restart Zabbix and run the setroubleshoot tool again.

Now we see a similar error as before, except this time Zabbix needs access to the zabbix_server_preprocessing.sock. Again, we can allow this access with the suggested commands.

And again, restart the Zabbix server.

Now things seem much happier.

Let’s run the setroubleshoot tool once more to ensure there are no more errors.

Now that the Zabbix server appears to be happy, be sure to set the server to start automatically.

So now that Zabbix server is running, we still need to finish the web frontend install. Before we’re able to connect to the UI, however, we’ll need to open the necessary ports in the Linux firewalld daemon. If we look at the ports currently open, we can see that neither http nor https are allowed.

Simply add the following service rules with firewall-cmd to allow http/https.

While we’re at it, let’s go ahead and add the firewall rules to allow the active and passive Zabbix agent checks:

Reload the firewalld daemon to apply the new rules.

We can now view the new rules in the allowed services and ports.

And, finally, restart Apache.

Zabbix Frontend Configuration

In a browser, connect to the frontend configuration UI at http://server/zabbix where “server” is the IP address of your Zabbix server. On the welcome screen, click “Next step” to continue.

Once all the prerequisites have been satisfactorily met, click “Next step” to continue.

Provide the database connection information, using the zabbix database user and password from above and click, “Next step” to continue.

Next, configure the Zabbix server details.

Review the installation summary for accuracy and click, “Next step” to finalize the install.

The Zabbix frontend installation should now be completed. Click, “Finish” to exit.

Log into the Zabbix Admin Console with the following credentials:

Username: Admin
Password: zabbix

Upon logging in, you’ll likely see an error indicating that Zabbix is still not properly running.

Let’s head back to see our old friend, the setroubleshoot tool, to see if SELinux is again the culprit.

Sure enough. As we can see from the log, now that things are up and running, httpd is being prevented from communicating on the Zabbix port 10051. It then clearly gives us some guidance on what we need to do to allow this behavior. Run the suggested commands as follows:

Now restart Apache and Zabbix.

After refreshing the Zabbix console, we can see that our error is now gone and the server appears to be functioning properly.

SSL Configuration

As a final step before we begin to configure our new Zabbix server, let’s generate a certificate and enable SSL. Of course, if your organization has it’s own PKI, or you purchase named or wildcard certificates for your domain, you’ll want to follow those processes rather than the one shown here.

The process detailed here will generate a self-signed certificate. Replace the information below with the relevant location, server name, domain, and IP information, as it relates to your organization. First, generate a CNF file from which to generate the certificate info.

Next, generate the certificate with OpenSSL.

Edit the Apache SSL configuration to use the newly created certificate and private key. Locate the section headed “<VirtualHost _default_:443>” and edit as follows:

Restart Apache after the SSL configuration has been saved.

You should now be able to reach your Zabbix server and login via SSL.

Conclusion

That sums up the Zabbix 4.0 LTS install on CentOS 7. You should now have a working Zabbix server and be ready to configure, add hosts, roll out agents and explore the various and sundry items that Zabbix can monitor.

Resizing the Linux Root Partition in a Gen2 Hyper-V VM

Resizing the Linux Root Partition in a Gen2 Hyper-V VM

Without a doubt, modern virtualization has changed the landscape of enterprise computing forever. Since virtual machines are abstracted away from the physical hardware, changes in compute, memory, and storage resources become mere clicks of a mouse. And, as hypervisors mature, many operations that were once thought of as out-of-band tasks, such as adding storage or even memory can now be done with little, or even zero downtime.

Hyper-V SCSI Disks and Linux

In many cases, hypervisors are backed by large storage area networks (SANs). This provides shared storage for hypervisor nodes that supports failover clustering and high availability. Additionally, it gives administrators the ability to scale the virtual environment, including the ability to easily add or expand storage on existing virtual servers. Microsoft’s Hyper-V 2012 introduced Generation 2 VMs, which extends this functionality. Among the many benefits of Gen2 VMs, was the ability to boot from a SCSI disk rather than IDE. This requires UEFI rather than a legacy BIOS, so it’s only supported among newer operating systems. Many admins I talk to think this is limited to Microsoft Server 2012 and newer, probably because of the sub-optimal phrasing in the Hyper-V VM creation UI that altogether fails to mention Linux operating systems.

The fact is, however, that many newer Linux OSes also support this ability, as shown in these tables from Microsoft.

More Disk, Please

Once you’ve built a modern Linux VM and you’re booting from synthetic SCSI disks rather than emulated IDE drives, you gain numerous advantages, not the least of which is the ability to resize the OS virtual hard disk (VHDX) on the fly. This is really handy functionality – after all, what sysadmin hasn’t had an OS drive run low on disk space at some point in their career? This is simply done from the virtual machine settings in Hyper-V Manager or Failover Cluster Manager by editing the VHDX.

Now, if you’re a Microsoft gal or guy, you already know that what comes next is pretty straightforward. Open the Disk Management MMC, rescan the disks, extend the file system, and viola, you now automagically have a bigger C:\ drive. But what about for Linux VMs? Though it might be a little less intuitive, we can still accomplish the same goal of expanding the primary OS disk with zero down time in Linux.

On-the-Fly Resizing

To demonstrate this, let’s start with a vanilla, Hyper-V Generation 2, CentOS 7.6 VM with a 10GB VHDX attached to a SCSI controller in our VM. Let’s also assume we’re using the default LVM partitioning scheme during the CentOS install. Looking at the block devices in Linux, we can see that we have a 10GB disk called sda which has three partitions – sda1, sda2 and sda3. We’re interested in sda3, since that contains our root partition, which is currently 7.8GB, as demonstrated here by the lsblk command.

Now let’s take a look at df. Here we can see an XFS filesystem on our 7.8GB partition, /dev/mapper/centos-root which is mounted on root.

Finally, let’s have a look at our LVM summary:

From this information we can see that there’s currently no room to expand our physical volume or logical volume, as the entirety of /dev/sda is consumed. In the past, with a Gen1 Hyper-V virtual machine, we would have had to shut the VM down and edit the disk, since it used an emulated IDE controller. Now that we have a Gen2 CentOS VM with a SCSI controller, however, we can simply edit the disk on the fly, expanding it to 20GB.

Once the correct virtual disk is located, select the “Expand” option.

Next, provide the size of the new disk. We’ll bump this one to 20GB.

Finally, click “Finish” to resize the disk. This process should be instant for dynamic virtual hard disks, but may take a few seconds to a several minutes for fixed virtual hard disks, depending on the size of the expansion and speed of your storage subsystem. You can then verify the new disk size by inspecting the disk.

OK, so we’ve expanded the VHDX in Hyper-V, but we haven’t done anything to make our VM’s operating system aware of the new space. As seen here with lsblk, the OS is indifferent to the expanded drive.

Taking a look at parted, we again see that our /dev/sda disk is still showing 10.7GB. We need to make the CentOS operating system aware of the new space. A reboot would certainly do this, but we want to perform this entire operation with no downtime.



Issue the following command to rescan the relevant disk – sda in our case. This tells the system to rescan the SCSI bus for changes, and will report the new space to the kernel without a restart.

Now, when we look at parted again, we’re prompted to move the GPT table to the back of the disk, since the secondary table is no longer in the proper location after the VHDX expansion. Type “Fix” to correct this, and then once again to edit the GPT to use all the available disk space. Once this is complete, we can see that /dev/sda is now recognized as 20GB, but our sda3 partition is still only 10GB.

Next, from the parted CLI, next use the resizepart command to grow the partition to the end of the disk.

Our sda3 partition is now using the maximum space available, 20.2GB. The lsblk command also now correctly reports our disk as 20GB.

But what about our LVM volumes? As suspected, our physical volumes, volume groups and logical volumes all remain unchanged.

We need to first tell our pv to expand into the available disk space on the partition. Do this with the pvresize command as follows:

Sure enough, our pv is now 18.8GB with 10.00GB free. Now we need to extend the logical volume and it’s associated filesystem into the free pv space. We can do this with a single command:

Looking at our logical volumes confirms that our root lv is now 17.80GB of the 18.80GB total, or exactly 10.0GB larger than we started with, as one would expect to see.

A final confirmation with the df command illustrates that our XFS root filesystem was also resized.

Conclusion

So there you have it. Despite some hearsay to the contrary, modern Linux OSes run just fine as Gen2 VMs on Hyper-V. Coupled with a SCSI disk controller for the OS VHDX, this yields the advantage of zero-downtime root partition resizing in Linux, though it’s admittedly a few more steps than a Windows server requires. And though Linux on Hyper-V might not seem like the most intuitive choice to some sysadmins, Hyper-V has matured significantly over the past several releases and is quite a powerful and stable platform for both Linux and Windows. And one last thing – when you run critically low on disk space on Linux, don’t forget to check those reserved blocks for a quick fix!

Silicon Power Bolt B10 USB SSD Review

Silicon Power Bolt B10 USB SSD Review

It’s no secret that the advent of solid state drives (SSDs) has irrevocably changed the technology landscape. Quite simply, SSDs eliminated what has been the largest bottleneck in computing performance for quite some time – the spinning disk. With SSDs, performance metrics on workstations such as latency, throughput and input/output operations per second (IOPS) are nearly irrelevant, as other system components such as the SATA interface become the limiting factor.

There are other attributes of SSDs that aren’t often emphasized, however, such as size, weight and portability. This is likely owing in large part to the fact that drives typically conform to a standard 2.5″ or 3.5″ form factor to maintain backwards compatibility. However, crack open a typical 2.5″ SSD and it becomes pretty obvious that SSDs could be significantly smaller than the standard dictates. And though hard disk drives (HDDs) have also packed more and more data onto their 2.5″ and 3.5″ platters, it is stated that SSD storage density surpassed that of HDDs in 2016, and is expected to continue to out-pace advances in HDD recording technologies.

Portable Drives

When I travel, I always like to have some sort of portable storage with me. And, to be frank, portable is a term that’s thrown around pretty loosely these days. My mobile phone is “portable,” yet it barely fits in my front jeans pocket, and I certainly can’t climb into my buddy’s Tundra with it in there. This, it seems, has been the unfortunate trend in portable devices these days – as performance demand increases, so does the size. Often times, I simply grab a USB thumb drive or two when I’m on the road. But when it comes to thumb drives, even the best performing of the bunch leave a little to be desired. I could carry a 2.5″ external drive in my laptop bag, I suppose, but the added bulk becomes cumbersome, especially when competing for space with a mouse, extra batteries, an assortment of cables, and the laptop’s AC adapter. What I really want is something small, light, and fast. I want something easy to carry, but not limited with respect to capacity or performance – you know, like an SSD.

Silicon Power Bolt B10

Having been a longtime fan of Silicon Power’s wallet-friendly line of SSDs for our lab server OS drives, I was delighted when Silicon Power sent me a Bolt B10 portable SSD to review. This is the first portable SSD I’ve intimately used and, spoiler alert, I won’t be going back to thumb drives when I travel any time soon. Now let’s dig into the details of this little gem.

Form Factor

Conceptually, one of the things I like most about portable SSDs is that we’re finally capable of breaking existing form factor molds. Before SSDs, 2.5″ and 3.5″ hard drives defined the size and shape of portable mass storage. This meant relatively large, clunky, and generally unattractive boxes. Since internal SSDs only conform to these form factors to remain relevant in server, desktop, and laptop applications, design can take a front row with portable SSDs. Portable SSDs tend to possess sleeker lines, smaller packages, and generally more attractive aesthetics. They don’t even have to be rectangular – some manufacturers like Silicon Power even offer round flavors of portable SSDs.

The Bolt B10 conforms to a traditional rectangular shape, though by no means is this a bad thing. The drive is understated, yet attractive, and it’s credit card sized form fits comfortably in the palm of your hand. The drive is featherweight at 25g, almost too light, though not cheap feeling, and the smooth plastic just feels right. It’s the kind of combination that makes you want to fidget with it, like the satisfying click of a well-made ball point pen. You can see here that the drive has a smaller footprint than a Logitech M325 wireless mouse.

Inside the drive you’ll find the PCB that occupies about half of the actual drive enclosure volume, which is to say it’s quite small. Directing the show, you’ll find the Phison PS3111-S11 controller, which gets generally favorable reviews. We’ve had nothing but luck with Phison-controlled SSDs, so we’ve got no complaints here. You can also see the 4 x 32GB NAND chips as well as a micro-USB 3.1 Gen1 connector soldered to the PCB.

Power, Heat & Noise

One of the numerous benefits of small form-factor 2.5″ hard drives is that their enclosures can be driven solely from USB power. The Bolt B10 is no exception. It’s a USB 3.1 Gen 1 device, but is also 2.0 backwards-compatible so it’s power draw cannot exceed the USB 2.0 specification of 5 power units of 100mA, or 500mA total. At 5V this equates to a maximum power consumption 2.5 W, though I suspect the B10 draws about half of that, even when continuously reading or writing.

In fact one of the more interesting use cases (for me) of portable hard drives is slinging lossless music around when I’m on the go. Specifically, I like to be able to plug a drive into an after market stereo head-unit with front-panel USB. Unfortunately, the front-panel USB just doesn’t deliver enough power to spin up most standard 2.5″ USB drives. The B10 works flawlessly in this application, giving you up to 512 GB of easily transportable lossless music for your commute.

Additionally, solid state drives generate less heat and noise than their spinning counterparts, as one might expect. The SP Bolt B10 makes no discernible noise during operation and the tiny case feels cool to the touch even after long continuous writes.

Specifications & Features

Included in the box is the Bolt B10 and a Micro USB 3.0 cable as shown here.

Now let’s take a look at the manufacturer specifications for the B10:

Power supply DC 5V
Cable Micro-B (B10) to Type-A (PC/NB)
Capacity 128GB, 256GB, 512GB
Dimensions 80.0 x 49.5 x 9.4mm
Weight 25g
Material Plastic
Color Black
Interface USB 3.1 Gen1 / USB 3.0, USB 2.0 compatible
Performance Read(max.)400MB/s
Performance Write(max.)400MB/s
Supported OS Windows 10/8.1/8/7/Vista/XP, Mac OS 10.5.x or later, Linux 2.6.31 or later
Operating Temperature 0℃~ 70℃
Storage Temperature -40℃~ 85℃
Certification CE/FCC/BSMI/Green dot/WEEE/RoHS/EAC/KCC/RCM
Warranty 3 years

Performance and Features

Ultimately, performance is probably what people care about most in a portable SSD. The USB interface has a long history of offering underwhelming performance. USB 2.0 offered pretty measly transfer rates of 480 Mbps or 60 MB/s. Due to bus limitations, real-world speeds were closer to 35 MB/s, however. Even in 2000 when USB 2.0 was introduced, an average spinning drive could easily saturate the USB link. It wasn’t until the advent of USB 3.0, nearly 10 years later, that the USB interface was no longer the bottleneck. With transfer speeds of 5 Gbps (625 MB/s), USB 3.0 suddenly made spinning drives feel slow, and the thought of portable SSDs began to make a lot of sense.

In this case, the Bolt B10 tested was a 128 GB model and testing was performed on a modest Dell Laptop: a Latitude E6430, Core i7-3720QM CPU @ 2.60 GHz, 8 GB RAM, Silicon Power S60 120 GB SSD. Given that the Bolt B10 has theoretical maximum throughput of 400 MB/s, we should not be bottlenecked by the USB 3.0 interface.

With the queue depth set to 4, ATTO Benchmarks showed write speeds very near the claimed 400 MB/s, peaking at nearly 360 MB/s, while read speeds exceeded the listed specifications, reaching speeds of approximately 430 MB/s.

CrystalDiskMark’s numbers weren’t quite as glowing, but were still quite good overall.

Some real-world file copies yielded satisfactory results. Large file copies were generally characterized by peaking at over 280MB/s then leveling out to ~130-150MB/s for the duration of the copy.

Small file copies can be quite taxing for any storage media. The results here were also on par with other similar drives we’ve tested. Here you see the copy of the WinSXS Folder – 5.96 GB (6,405,198,876 bytes) containing 76,348 Files, 23,163 Folders.

Finally, Silicon Power lists the drive’s features as follows:

  • Ultra-compact and lightweight for great portability
  • Clean and smooth exterior design
  • Large storage capacity of up to 512GB
  • Superfast transfer rates of up to 400MB/s read & write speed*
    *The transmission speed will vary depending on system performance, such as hardware, software, file system, usage and product capacity. Speeds are tested by Silicon Power with FAT32 (for cross-platform sharing) or NTFS (for single file over 4GB) file formats using CDM or ATTO tests.
  • Supports LDPC error correction and Wear Leveling
  • Free download of SP Widget software for data backup & restore, AES 256-bit encryption, and cloud storage
  • One thing to note is that out of the box the Bolt B10 was formatted to FAT32, which is an interesting choice. As such, I could not initially copy files larger than 4 GB to the drive. Now to someone who’s been in IT for 20 years, this isn’t a big deal, and a quick reformat to NTFS resolved the issue. However, I can easily see how this might confuse someone a bit less technology savvy. Additionally, one of my pet peeves about many external hard drives are the hordes of autorun software that come pre-loaded. Most people simply want to drag and drop files to their USB drives, so this software is ordinarily just a nuisance. On a positive note, the SP Bolt B10 contains very little on the drive out of the box. In fact, the only files present on the drive were there to remind you to register your product with Silicon Power.

    Conclusion

    It should be no surprise that SSDs are now the logical choice when it comes to no-compromise portable storage. And though you’re certainly not going to tote around 4TB SSDs anytime soon (unless you have really deep pockets), affordable, portable SSDs are now large enough to meet most users’ needs. Silicon Power offers just such a drive in the Bolt B10. Are there faster portable SSDs out there? Sure, at least on paper. Considering that you’ll be tossing this in your bag and possibly leaving it on the table at the coffee shop, I’m not sure I can justify the extra cash for a few arbitrary MB/s. Additionally, it seems that many manufacturers rate their products in the lab, and under conditions that are hard to replicate in the real-world. It’s been my experience, however, that Silicon Power’s products usually meet or exceed claimed specifications. Frankly, realistic product specifications are a breath of fresh air, and make you feel like you’re getting your money’s worth, all while patronizing a company that clearly wants to earn your trust.

Joining a Windows Domain with Centrify Express

Joining a Windows Domain with Centrify Express

As you’ve seen us mention in our Linux File Servers in a Windows Domain article, Linux systems have become an omnipresent fixture of the IT landscape, even in companies that are heavily invested in Windows infrastructure. But one challenge to managing such infrastructure diversity is maintaining standardization among disparate systems. Ask any IT Admin who’s managed Linux systems over the last 20 years and you’ll likely hear numerous stories about rogue servers, shared accounts, a lack of password complexity enforcement, and a plain lack of standardization. Things have changed a lot over the past decade, however. With a general shift to virtualization, as well as provisioning tools from Ubuntu and Red Hat, and third-party tools like Ansible, system standardization on Linux has become easier than ever.

Yet another complication of a diverse environment is identity management. Especially with the ever-looming threat of virtual machine sprawl, managing disparate systems and their respective logins can be tiresome at best, and a downright security risk at worst. Thus, when considering systems and applications, a good engineer should always ask questions about identity management and integration with Active Directory, LDAP, or other directory service. Along the way Samba has offered some ability to integrate Linux systems into Active Directory, but it wasn’t always easy to implement, and identity management wasn’t Samba’s primary focus. Today, however, there are now dedicated tools to manage identities across numerous systems, including Linux, UNIX, Macs, and beyond.

One such tool that simplifies and unifies identity management across multiple platforms is Centrify. Centrify was founded in 2004 and offers software designed to thwart the number one point of entry in a data breach – compromised credentials. To say it’s a trusted platform would be an understatement – Centrify claims over half of the Fortune 100 trusts some form of their identity and access management to Centrify.

Here at Teknophiles, we look for a couple of things in a software package for use in our lab:

  1. Does the software offer Enterprise-level performance and functionality?
  2. Does the vendor provide IT professionals with an inexpensive (or free) means to test or use the software, at least in some limited capacity?

We’re not a big fan of 30-day trials at Teknophiles, because as any IT Pro knows, it’s really tough to dig into a software package and learn the ins and outs in such a limited time frame, especially with a day job. And in a lab environment, it’s almost mandatory to be able to leave a piece of hardware or software in place for an extended length of time, so that it can be tested in future configurations and scenarios (Take note, Microsoft, Re: TechNet!). We also understand that software companies are in business to make money, which is why we like the, “Free, but limited” model. In the, “Free, but limited” approach, software may be limited to a certain scope of install, number of nodes, or reduced feature set. There are lots of great examples that follow this model – Nagios Core, Thycotic Secret Server, Sophos UTM, among others, and Centrify is no exception. To get the full feature set, one must upgrade to the licensed version. Typically, there’s an easy upgrade path to the licensed version, which gives IT Admins the added confidence that they can stand up a piece of software and, if they and their superiors decide it’s a good fit, simply upgrade without standing up a completely new system.

I happen to use Centrify daily at my full-time job. It’s become instrumental in managing and standardizing access to the numerous Oracle Linux and RHEL systems we deploy. Given my experience with Centrify in the Enterprise environment, I was delighted to learn that they offer a limited package, called Centrify Express, that allows for installation on 200 servers, albeit with a reduced feature set. Here’s a table with a brief comparison of features between Centrify Infrastructure Services and Centrify Express: Reasons to Upgrade

Before You Start

In this article, we’ll briefly cover the installation of the Centrify Express Agent on CentOS 7. Before we dig in, however, there are a couple of things you need to make sure you have in order beforehand.

First, it probably goes without saying, but you need a working Active Directory environment with at least one functioning Domain Controller. You also need the credentials for a user who has the ability to add systems to the domain. It’s probably easiest if you use an account that is a Domain Admin or has similarly delegated permissions within Active Directory.

Second, as we’ve mentioned previously, DNS is a critical component of a functional Active Directory implementation. Be sure you have good FQDN (server.domain.com) resolution between the Domain Controllers and the system you wish to become a domain member, as well as name resolution from the Linux member server to the Domain Controllers (DC01.domain.com, DC02.domain.com, etc.)

Finally, give yourself at least one local Linux account with sudo access that is NOT the same as any AD account. Should you at some point find yourself unable to log into the server via Active Directory, you can use this account as a fallback.

WARNING

If the local account has the same name as an AD account, the system will assume you are trying to use the AD account and you may be unable to log in. If this is the only local account with sudo access, you could find yourself without the ability to administer the server!

With that out of the way, let’s move on to the fun parts.



Staging the Centrify Express Installation Files

Begin by downloading Centrify Express for the proper flavor of Linux here. You’ll have to fill out a short form to gain access to the installers, but it’s free and there’s no commitment. In fact, unlike some other software vendors, I haven’t been plagued by sales calls, either.

Once, you have the proper installer downloaded, simply SFTP the file to the server you wish to become a domain member. On the server, create a folder to extract the installation files.

Next, extract the installation files as follows.

Now, simply begin the installation by executing the install.sh script.

Select X for the Centrify Express installation.

Select Y to continue to install in Express mode.

Select Y to verify the AD environment. This is a good idea, as it will flag any major issues prior to attempting the domain join.

Enter the domain you wish to join.

Confirm you want to join an Active Directory Domain by selecting Y.

Again, enter the name of the domain you wish to join.

Next, enter the username and password of an admin with permission to join systems to the domain. It doesn’t have to be a Domain Admin, but it using a Domain Admin account may simplify permissions troubleshooting:

Verify the computer name and the container DN within Active Directory. In most cases the Computers container will suffice, as you can always move the machine later for organizational purposes. If you choose, you can also specify a different container, as shown here.

Now, enter the name of the Domain Controller you wish to use to join the domain. Typically this can be left as auto detect, but you can also specify a DC.

Choose whether you want the system to reboot upon completion of the install and domain join. Though this is not required, other services may need to be restarted for full integration.

Finally, confirm the options you provided and select Y to proceed.

View the output of the installation summary and resolve any issues. In this case you can see we had one warning regarding SSH configuration. Any issues may need to be addressed for complete integration.

Before logging out as root, be sure to add domain admins or other Linux administrator AD group to the sudoers file, by adding the following line.

You should now be able to log in via a domain account. Use only the user’s shortname (not FQDN).

You can quickly confirm that your AD account is working properly by viewing your user’s domain group memberships.

That’s it for the install! We hope to provide additional Centrify walk-throughs in a future article (Centrify Samba/PuTTY), but this should quickly get you up and running with single sign-on for your domain-integrated Linux servers. As you can see, Centrify provides a neat and tidy package to manage identities across multiple server platforms in a Windows Domain. It’s rock-solid, and right at home in a small home lab or large Enterprise. We’ve been using it for years now without issue in both of environments, and hope you’ll find it as useful as we have.

Network Configuration in RHEL/CentOS Linux 7

Network Configuration in RHEL/CentOS Linux 7

Although numerous and significant changes have been introduced in recent versions of Red Hat Enterprise Linux/CentOS, one of the most hotly contested topics seems to center around network configuration. In older versions of RHEL/CentOS, the default means of configuring network interfaces was via network scripts and use of the Network Administration Tool, which is now deprecated. Additionally, the well-known net-tools are also considered obsolete, replaced by the less-familiar ip commands. From RHEL/CentOS 6 onward, emphasis is now placed on using NetworkManager for network configuration, as well as the aforementioned ip command set. As of RHEL/CentOS 7, NetworkManager is considered the default method for managing network configurations, though the RHEL development teams paid careful attention to ensure that the network scrips would still function, and that they play nice with NetworkManager. Note this passage from the RHEL Networking Guide describing the relationship between NetworkManager and the older networking scripts:

“Note that in Red Hat Enterprise Linux 7, NetworkManager is started first, and /etc/init.d/network checks with NetworkManager to avoid tampering with NetworkManager’s connections. NetworkManager is intended to be the primary application using sysconfig configuration files and /etc/init.d/network is intended to be secondary, playing a fallback role.”

Old habits die hard, however, and you’ll likely see a number of articles across the Internet that detail methods to workaround NetworkManager, describe NetworkManager as, “the devil,” or “the plague,” and claim that most of the difficulties that arise with network configuration in RHEL/CentOS 7 are due to NetworkManager. While I haven’t seen any catastrophic effects from NetworkManager’s use, if it isn’t properly configured or if you aren’t well versed in its syntax, it can certainly be confusing and produce unintended results. In this article we’ll attempt to reserve judgement, and walk you through both methods of network configuration and let you decide which you prefer. First, let’s begin with NetworkManager and some of the tools used for configuration.

NetworkManager

NetworkManager is described as a “dynamic network control and configuration daemon.” Though the traditional ifcfg scripts are supported, NetworkManager is intended to the be the default mechanism for network configuration in RHEL/CentOS 7. It offers fairly extensive control of a variety of network interface types, including more modern connectivity such as Wi-Fi, mobile broadband, infiniband, as well as traditional Ethernet connections. At Teknophiles, we primarily deal with server environments in which our primary concern is the latter – traditional static Ethernet configurations. As such, much of the added functionality that NetworkManager offers will be superfluous to our discussion here. More detail on NetworkManager can be found here.

nmcli

In a server environment, NetworkManager is primarily managed by two user interfaces, nmcli (command-line interface) and nmtui (text user interface). We’ll first detail nmcli, since that’s what many administrators will prefer. Additionally, we feel strongly that understanding the cli is important since it helps to understand what is being invoked behind the scenes when using the tui. In its simplest form nmcli produces basic interface information and DNS configuration:

More detail can be had by looking at a specific network device. This command is shorthand for “nmcli device show.” Note the connection name, “Wired connection 1”.

Using the connection name, we can also look at the connection detail, which produces quite a bit of output:

This is a pretty typical configuration for a RHEL/CentOS box in its default configuration. From this output we can determine a good bit of useful information which we’ll need to configure a static IP on our server. Note that these attributes all take the format “setting.property”, with the property value listed in the right-hand column. We can see here that the connection has a human-readable ID, called “connection.id,” which is arguably somewhat unwieldy:

We can also see that the interface device is eth0, the connection type is Ethernet (802-3-ethernet), and this configuration was set by DHCP (auto).

In a typical server configuration, you’ll likely want to change several of these parameters to match your environment or standard practices. We can use the nmcli command to bring up an interactive command line tool to edit an existing connection.

The “?”, “help”, or “h” command will bring up a quick reference of available commands.

First, let’s create a more friendly connection name. Since we know this is a “connection” setting, type “goto connection” to enter the connection setting menu. Typing “print” or “p” will display the current values of this setting. The print command is contextual – it will display the values for your current location, in this case the connection menu.

We can now change connection.id value with the “set” command. Since the commands are contextual, we simply need to invoke the command as shown below. If we were not in the connection sub-menu, we would have to use the full setting.property format, connection.id, in our command.

We can see that our connection.id is now static-eth0, which is much easier to type or call in scripts, since it now lacks spaces and is much less generic. Before we continue, however, there’s one more important step. If you’ve ever configured a Cisco or other enterprise switch, you know that simply updating a config and walking away can be disastrous if you do not commit the running configuration. NetworkManager’s interactive configuration works much the same way – though you’ll see the changes to your interfaces immediately, those settings will not survive a restart unless you save the configuration. This is done with a simple “save” command.

Now let’s activate the changes and take a look at the results by looking at the connection properties again.

If you’d prefer not to use the interactive editor, this can also be accomplished via the nmcli in a single command as follows.

Let’s move on to setting the interface to use a static IPv4 address. We’ll also disable IPv6 with this command as well.

As you can see, nmcli is quite powerful. With a single command, we can very quickly modify our network configuration, without the need for modifying networks scripts or other complicated configurations.



nmtui

The nmtui utility is a text user interface designed to perform many of the same functions as nmcli. Though some Linux sysadmins distrust UIs (we’ve seen bugginess in some UIs ourselves), nmtui provides a quick way to configure a network interface via NetworkManager.

First, select the option to, “Edit a configuration,” then select, “Wired connection 1” from the list of Ethernet connections.

Next, edit the profile name, then highlight <Show> next to = IPv4 CONFIGURATION and press ENTER.

Set the following items, according to your environment:

Highlight <OK> when done and press ENTER.

Now go back to the initial NetworkManager TUI menu and select “Quit” to exit to the shell. Activate the new configuration as follows.

Finally, take a quick look at /etc/resolv.conf. You’ll notice that NetworkManager was kind enough to edit these values for you.

No School Like the Old School

As we previously stated, there’s still a great deal of loyalty to the old network scripts method for configuring networking in RHEL/CentOS 7. Since the RHEL devs made sure that network scripts still function as expected, there’s a fairly straightforward means of circumventing NetworkManager and using the network scripts. This is a boon to many sysadmins, who already have well-documented and streamlined processes in place for network configuration. To use the network scripts for configuring networking in RHEL/CentOS 7, first disable the NetworkManager.service.

NOTE

You may also elect to use the “systemctl mask” command, but this might make swapping back and forth to test each method a bit less trivial. If you’ve settled on using the network scripts for network management, we do recommend using “systemctl mask NetworkManager.service” in your final configuration.

Now that the NetworkManager daemon won’t restart upon reboot and won’t interfere with our manual configuration, we can configure our eth0 interface by creating or modifying the appropriate network script.

NOTE

Make sure this is the ONLY network script that references this device and/or MAC address. This is especially relevant if NetworkManager has been previously running things and you have made manual changes.

After saving the configuration file, restart the network service.

Alternatively, bounce the network interface.

net-tools vs. ip

Lastly, we’ll complete the discussion by looking at the old net-tools vs the ip command. There’s really not much to say here, as the net-tools have been deprecated and haven’t been updated in ages. Yet I’m sure there are those to will cling to the old tools for quite some time – I certainly find myself still relying on ifconfig and route quite a bit. On the surface each tool offers much of the same information, though truth be told, to my over-40 eyes, the old tools do provide more pleasing output.

Conclusion

As with most things Linux, there’s more than one way to skin the proverbial cat. Network configuration is no exception. Sometimes it’s difficult to let go of old methodologies and embrace the new, especially when you’ve been doing something the same way for many years. I think this is probably the case with some of the bad press surrounding NetworkManager, especially in server environments. On the other hand, older, simpler methods are sometimes faster and less error-prone than newer, less-familiar ones. In the end, it’s up to you decide which method you prefer, but both network scripts and NetworkManager should provide you the tools you need to successfully manage your Linux network configurations.

Disk Pooling in Linux with mergerFS

Disk Pooling in Linux with mergerFS

Remember the old days when we used to marvel at disk drives that measured in the hundreds of megabytes? In retrospect it seems funny now, but it wasn’t uncommon to hear someone mutter, “Man, we’ll never fill that thing up.”

If you don’t remember that, you probably don’t recall life before iPhones either, but we old timers can assure you that, once upon a time, hard drives were a mere fraction of the size they are today. Oddly enough, though, you’ll still hear the same old tripe, even from fellow IT folks. The difference now, however, is that they’re holding a helium-filled 10TB drive in their hands. But just like yesteryear, we’ll fill it up, and it’ll happen faster than you think.

In the quest to build a more scalable home IT lab, we grew tired of the old paradigm of building bigger and bigger file servers and migrating hordes of data as we outgrew the drive arrays. Rather than labor with year-over-year upgrades, we ultimately arrived at a storage solution that we feel is the best compromise of scalability, performance, and cost, while delivering SAN-NAS hybrid flexibility.

We’ll detail the full storage build in another article, but for now we want to focus specifically on the Linux filesystem that we chose for the NAS component of our storage server: mergerFS.

What is mergerFS?

Before we get into the specifics regarding mergerFS, let’s provide some relevant definitions:

Filesystem – Simply put, a filesystem is a system of organization used to store and retrieve data in a computer system. The filesystem manages the space used, filenames, directories, and file metadata such as allocated blocks, timestamps, ACLs, etc.

Union Mount – A union mount is a method for joining multiple directories into a single directory. To a user interfacing with the union, the directory would appear to contain the aggregate contents of the directories that have been combined.

FUSE – Filesystem in Userspace. FUSE is software that allows non-privileged users to create and mount their own filesystems that run as an unprivileged process or daemon. Userspace filesystems have a number of advantages. Unlike kernel-space filesystems, which are rigorously tested and reviewed prior to acceptance into the Linux Kernel, userspace filesystems, since they are abstracted away from the kernel, can be developed much more nimbly. It’s also unlikely that any filesystem that did not address mainstream requirements would be accepted into the kernel, so userspace filesystems are able to address more niche needs. We could go on about FUSE, but suffice it to say that userspace filesystems provides unique opportunities for developers to do some cool things with filesystems that were not easily attainable before.

So what is mergerFS? MergerFS is a union filesystem, similar to mhddfs, UnionFS, and aufs. MergerFS enables the joining of multiple directories that appear to the user as a single directory. This merged directory will contain all of the files and directories present in each of the joined directories. Furthermore, the merged directory will be mounted on a single point in the filesystem, greatly simplifying access and management of files and subdirectories. And, when each of the merged directories themselves are mount points representing individual disks or disk arrays, mergerFS effectively serves as a disk pooling utility, allowing us to group disparate hard disks, arrays or any combination of the two. At Teknophiles, we’ve used several union filesystems in the past, and mergerFS stands out to us for two reasons: 1) It’s extremely easy to install, configure, and use, and 2) It just plain works.

An example of two merged directories, /mnt/mergerFS01 and /mnt/mergerFS02, and the resultant union.

mergerFS vs. LVM

You might be asking yourself, “why not just use LVM, instead of mergerFS?” Though the Logical Volume Manager can address some of the same challenges that mergerFS does, such as creating a single large logical volume from numerous physical devices (disks and/or arrays), they’re quite different animals. LVM sits just above the physical disks and partitions, while mergerFS sits on top of the filesystems of those partitions. For our application, however, LVM has one significant drawback: When multiple physical volumes make up a single volume group and a logical volume is carved out from that volume group, it’s possible, and even probable, that a file will be comprised of physical extents from multiple devices. Consider the diagram below.

The practical concern here is that should an underlying physical volume in LVM fail catastrophically, be it a single disk or even a whole RAID array, any data on, or that spans, the failed volume will be impacted. Of course, the same is true of mergerFS, but since the data does not span physical volumes, it’s much easier to determine which files are impacted. There’s unfortunately no easy way that we’ve found to determine which files are located on which physical volumes in LVM.



Flexibility & Scalability

As we’ve alluded to a few times, mergerFS doesn’t care if the underlying physical volumes are single disks or RAID arrays, or LVM volumes. Since the two technologies operate at different levels with respect to the underlying disks, nothing prevents the use of LVM with mergerFS. In fact, we commonly use the following formula to create large mergerFS disk pools: multiple mdadm 4-Disk RAID10 arrays > LVM > mergerFS. In our case, the LVM is usually just used for management, we typically do not span multiple physical volumes with any volume groups, though you easily could.

This gives you incredible flexibility and scalability – want to add an individual 5400 RPM 4TB disk to an existing 12TB RAID6 array comprised of 6+2 7200 RPM 2TB drives, for 16TB total? No problem. Want to later add an LVM logical volume that spans two 8TB 2+2 RAID10 arrays for another 16TB. MergerFS is fine with that, too. In fact, mergerFS is completely agnostic to disk controller, drive size, speed, form factor, etc. With mergerFS, you can grow your storage server as you see fit.

mergerFS & Your Data

One interesting note about mergerFS is that since it is just a proxy for your data, it does not manipulate the data in any way. Prior to being part of a mergerFS union, each of your component disks, arrays, and logical volumes will already have a filesystem. This makes data recovery quite simple – should a mergerFS pool completely crash (though, unlikely), just remove the component storage devices, drop them a compatible system, mount as usual, and access your data.

What’s more, you can equally as easily add a disk to mergerFS that already has data on it. This allows you to decide at some later point if you wish to add an in-use disk to the mergerFS pool (try that with LVM). The existing data will simply show up in the mergerFS filesystem, along with whatever data is on the other volumes. It just doesn’t get any more straightforward!

mergerFS & Samba

As we stated earlier, we selected mergerFS for the NAS component of our Teknophiles Ultimate Home IT Lab storage solution. Since this is a NAS that is expected to serve files to users in a Windows Domain, we also run Samba to facilitate the Windows file sharing. Apparently, there are rumblings regarding issues with mergerFS and Samba, however, according to the author of mergerFS, this is likely due to improper Samba configuration.

Here at Teknophiles, we can unequivocally say that in server configurations based on our, “Linux File Servers in a Windows Domain,” article, Samba is perfectly stable with mergerFS. In fact, in one mergerFS pool, we’re serving up over 20TB of data spread over multiple mdadm RAID arrays. The server in question is currently sitting at 400 days of uptime, without so much as a hiccup from Samba or mergerFS.

Installing mergerFS

OK, so that’s enough background, now let’s get to the fun part. To install mergerFS, first download the latest release for your platform. We’re installing this on an Ubuntu 14.04 LTS server, so we’ll download the Trusty 64-bit .deb file. Once downloaded, install via the Debian package manager.

Creating mergerFS volumes

Now we’re going to add a couple of volumes to a mergerFS pool. You can see here that we have a virtual machine with two 5GB virtual disks, /dev/sdb and /dev/sdc. You can also see that each disk has an ext4 partition.

Next, create the mount points for these two disks and mount the disks in their respective directories.

Now we need to create a mount point for the union filesystem, which we’ll call ‘virt’ for our virtual directory.

And finally, we can mount the union filesystem. The command follows the syntax below, where <srcmounts> is a colon delimited list of directories you wish to merge.

mergerfs -o<options> <srcmounts> <mountpoint>

Additionally, you can also use globbing for the source paths, but you must escape the wildcard character.

There are numerous additional options that are available for mergerFS, but the above command will work well in most scenarios. From the mergerFS man page, here’s what the above options do:

defaults: a shortcut for FUSE’s atomic_o_trunc, auto_cache, big_writes, default_permissions, splice_move, splice_read, and splice_write. These options seem to provide the best performance.

allow_other: a libfuse option which allows users besides the one which ran mergerfs to see the filesystem. This is required for most use-cases.

use_ino: causes mergerfs to supply file/directory inodes rather than libfuse. While not a default it is generally recommended it be enabled so that hard linked files share the same inode value.

fsname=name: sets the name of the filesystem as seen in mount, df, etc. Defaults to a list of the source paths concatenated together with the longest common prefix removed.

You can now see that we have a new volume called, “mergerFS,” which is the aggregate 10GB, mounted on /mnt/virt. This new mount point can be written to, used in local applications, or served up via Samba just as any other mount point.

Other Options

Although getting a little into the weeds, it’s worth touching on an additional option in mergerFS that is both interesting and quite useful. The FUSE function policies determine how a number of different commands behave when acting upon the data in the mergerFS disk pool.

func.<func>=<policy>: sets the specific FUSE function’s policy. See below for the list of value types. Example: func.getattr=newest

Below we can see the FUSE functions and their category classifications, as well as the default policies.

Category FUSE Functions
action chmod, chown, link, removexattr, rename, rmdir, setxattr, truncate, unlink, utimens
create create, mkdir, mknod, symlink
search access, getattr, getxattr, ioctl, listxattr, open, readlink
N/A fallocate, fgetattr, fsync, ftruncate, ioctl, read, readdir, release, statfs, write
Category Default Policy
action all
create epmfs
search ff

To illustrate this a bit better, let’s look at an example. First, let’s consider file or directory creation. File and directory creation fall under the “create” category. Looking at the default policy for the create category we see that it is called, “epmfs.” From the man pages, the epmfs policy is defined as follows:

epmfs (existing path, most free space)
Of all the drives on which the relative path exists choose the drive with the most free space. For create category functions it will exclude readonly drives and those with free space less than min-freespace. Falls back to mfs.

Breaking this down further, we can see that epmfs is a “path-preserving policy,” meaning that only drives that have the existing path will be considered. This gives you a bit of control over where certain files are placed. Consider, for instance if you have four drives in your mergerFS pool, but only two of the drives contain a directory called /pictures. When using the epmfs policy, only the two drives with the pictures directory will be considered when you copy new images to the mergerFS pool.

Additionally, the epmfs policy will also serve to fill the drive with the most free space first. Once drives reach equal or near-equal capacities, epmfs will effectively round-robin the drives, as long as they also meet the existing path requirement.

There are a number of equally interesting policies, including ones that do not preserve paths (create directories as needed), fill the drive with the least free space (i.e. fill drive1, then drive2, etc.), or simply use the first drive found. Though the defaults will generally suffice, it’s a good idea to become familiar with these policies to ensure that your mergerFS configuration best suits your needs.



Adding Volumes

Similar to creating a mergerFS pool, adding disks is also quite simple. Let’s say, for instance, you want to also use the space on your existing operating system disk for the mergerFS pool. We can simply create another directory in the root filesystem to use in the union. We created ours in /mnt for demonstration purposes, but your home directory might equally suit.

Next, we unmount our existing mergerFS pool and remount it including the new directory.

Notice now our total mergerFS volume space is 19GB – the original 10GB from our two 5GB disks, plus the 9GB from /dev/sda2. And now, since we now have truly disparate volumes, let’s test our default epmfs policy. Start by creating three files in our mergerFS pool mount:

Based on our expectations of how epmfs works, we should see these files in the /mnt/mergerFS00 folder, since /dev/sda1 has the most free space.

Sure enough, this appears to work as we anticipated. Now let’s create a folder on one disk only.

Replicating our previous experiment, we’ll create a few more files, but this time in the pics directory in our mergerFS pool.

Since the epmfs policy should preserve the path pics/, and that path only exists on /mnt/mergerFS01, this is where we expect to see those files.

Removing Volumes

Removing a volume from the mergerFS pool follows the same procedure as adding a drive. Simply remount the pool without the path you wish to remove.

Notice now file1, file2, and file2 are no longer present, since they were located on /dev/sda2, which has been removed. Additionally, our total space is now back to its previous size.

mergerFS & fstab

Typically, you’re going to want your mergerFS pool to be persistent upon reboot. To do this we can simply leverage fstab as we would for any mount point. Using our example above, fstab should follow the following format:

Performance Considerations

We would be remiss to end this article without disucssing the possible performance implications of mergerFS. As with any disk pooling utility, a possible weakness of this type of this configuration is a lack of striping across drives. In RAID or LVM configurations, striping may be used to take advantage of simultaneous I/O and throughput of the available spindles. RAID level, array size, and LVM configuration play dramatically into exactly what this looks like, but even a 6+2 RAID6 array with commodity drives can achieve read speeds that will saturate a 10Gbps network. Use LVM to stripe across multiple arrays, and you can achieve stunning performance. If you’re only using single disks in your mergerFS pool, however, you’ll always be limited to the performance of a single drive. And maybe that’s OK, depending on your storage goals. Of course, careful planning of the disk subsystem and using disk arrays in your mergerFS pool can give you the best of both worlds – excellent performance and the flexibility and scalability of disk pooling.

Lastly, it’s worth noting that mergerFS is yet another layer on top of your filesystems and FUSE filesystems by their very nature add some overhead. This overhead is generally negligible, however, especially in low to moderate I/O environments. You might not want to put your busy MySQL database in userspace, but you’ll likely not notice the difference in storing your family picture albums there.

Reclaim Linux Filesystem Reserved Space

Reclaim Linux Filesystem Reserved Space

As IT Pros, we have a myriad of tools available to us to configure and tweak and tune the systems we manage. So much so, there are often everyday tools right under our noses that might have applications we may not immediately realize. In a Linux environment, tune2fs is an indispensable tool, used to tune parameters on ext2/ext3/ext4 filesystems. Most Linux sysadmins that have used mdadm software RAID will certainly recognize this utility if they’ve ever had to manipulate the stride size or stripe width in an array.

tune2fs

First, Let’s take a look at the disks on an Ubuntu file server so we can see what this tool does.

Now, we can use the tune2fs with the -l option to list the existing parameters of the filesystem superblock on /dev/sdb1.

Reserved Blocks?

As you can see, there are a number of parameters from the filesystem that we can view, including a number that can be tuned with tune2fs. In this article however, we’re going to focus on a rather simple and somewhat innocuous parameter – reserved block count. Let’s take a look at that parameter again:

At first glance, it isn’t obvious what this parameter means. In fact, I’ve worked with Linux sysadmins with years of experience that weren’t aware of this little gem. To understand this parameter, we probably have to put it’s origins in a bit of context. Once upon a time, SSDs didn’t exist, and no one knew what a terabyte was. In fact, I remember shelling out well north of a $100 for my first 20GB drive. To date myself even further, I remember the first 486-DX PC I built with my father in the early ’90s, and it’s drive was measured in megabytes. Crazy, I know. Since drive space wasn’t always so plentiful, and the consequences of running out of disk space on the root partition in a Linux system are numerous, early filesystem developers did something smart – they reserved a percentage of filesystem blocks for privileged processes. This ensured that even if disk space ran precariously low, the root user could still log in, and the system could still execute critical processes.

That magic number? Five percent.

And while five percent of that 20GB drive back in 1998 wasn’t very much space, imagine that new 4-disk RAID1/0 array you just created with 10TB WD Red Pros. That’s five percent of 20TB of usable space, or a full terabyte. You see, though this was likely intended for the root filesystem, by default this setting applies to every filesystem created. Now, I don’t know about you, but at $450 for a 10TB WD Red Pro, that’s not exactly space I’d want to throw away.



We Don’t Need No Stinking Reserved Blocks!

The good news, however, is that space isn’t lost forever. If you forget to initially set this parameter when you create the filesystem, tune2fs allows you to retroactively reclaim that space with the -m option.

Here you can see we’ve set the reserved blocks on /dev/sdb1 to 0%. Again, this isn’t something you’d want to do on a root filesystem, but for our “multimedia” drive, this is fine – more on that later. Now, let’s look at our filesystem parameters once again.

Notice now that our reserved blocks is set to zero. Finally, let’s have a look at our free disk space to see the real world impact. Initially, we had 50GB of 382GB free. Now we can see that, although neither the size of the disk nor the amount of used space has changed, we now have 69GB free, reclaiming 19GB of space.

Defrag Implications

Lastly, I’d be remiss if I didn’t mention that there’s one other function these reserved blocks serve. As always, in life there’s no such thing as a free lunch (or free space, in this case). The filesystem reserved blocks also serve to provide the system with free blocks with which to defragment the filesystem. Clearly, this isn’t something you’d want to do on a filesystem that contained a database, or in some other situation in which you had a large number of writes and deletions. However, if like in our case, you’re dealing with mostly static data, in a write once, read many (WORM) type configuration, this shouldn