Installing Zabbix 4.0 LTS on CentOS 7

Installing Zabbix 4.0 LTS on CentOS 7

When it comes to enterprise monitoring solutions, there are a myriad of options to choose from, both paid and free and open-source software (FOSS). Here at Teknophiles, we’ve used just about all of them, and we can tell you that many are good, some are even great, but none are perfect. Even at some of the higher price points, we usually find something we don’t care for. Generally speaking, paying good money for monitoring software will get you support, and perhaps some ease of configuration that open source solutions do not offer.

That said, there are numerous free and open-source monitoring options that get the job done quite well. After using Nagios for many years, we’ve recently become quite enamored with Zabbix. Though not trivial to configure, especially in large environments (what monitoring solution is?), Zabbix is stable, polished and quite extensible. In this article – the first of several in a series on installing and configuring Zabbix – we’ll walk through a typical Zabbix install on Linux, using the latest long-term release from Zabbix, so you can take it for a test drive.

Zabbix Server Sizing

To get started, we’ll first assume you have a running CentOS 7 box, either a virtual or physical machine. With respect to hardware requirements, lab/test/home environments shouldn’t require more than 1-2 CPU cores and 2-4 GB of RAM. We’re currently monitoring over 3600 items, with over 2000 triggers on a virtual server with 1 CPU core and 2GB of RAM, and the server rarely shows any significant resource utilization.

Though disk usage depends on many factors, the Zabbix database will be the primary offender when it comes to disk consumption. Once Zabbix is installed and configured, the application files will change very little in terms of size, so whether or not the database is local or remote will greatly impact disk sizing for the Zabbix server. Regardless of the database location, make sure you have enough space for the database to grow as you accumulate historical data. This will be influenced by not only the number of hosts and items monitored, but also historical and trend storage periods. As a general starting point for a lab server with a local database, a root partition of 15-20 GB should suffice. You can see here that with just over 3600 items, our lab database is chewing up approximately 600 MB/month. This will likely stabilize at some point after all data retention periods are reached, but should be a consideration when sizing the disks.

Do some due diligence and size your Zabbix environment appropriately up front, however. Though this can depend on many factors, such as whether your database is local or remote, whether you choose to use MySQL InnoDB vs. PostgreSQL, the number of hosts you wish to monitor and whether or not you choose to use Zabbix Proxies, it’s much easier to provision resources earlier in the process than later, especially on physical hardware. Also, keep in mind that in medium to large installations (500-1000 hosts), disk I/O will also start become an important factor, along with CPU and memory considerations. Plan the disk subsystem accordingly. Some general sizing recommendations from the folks at Zabbix can be found here.

SELinux

For the purposes of this article, we’re also going to assume you’re a responsible Linux sysadmin and will be installing Zabbix with SELinux in enforcing mode. We strongly recommend that you leave SELinux this way. We’re aware that SELinux can be challenging and the knee-jerk reaction by many is to disable SELinux – we’ve been tempted ourselves at times. SELinux naturally adds a few steps to the process, but it’s completely manageable with the tools available, and will ultimately leave you with a better security stance. You can check the current status of SELinux as follows:

Prerequisites

The Zabbix server installation requires several prerequisite components to function – an Apache web server, PHP, and a database server. First, we’ll install the standard Apache httpd server with mod_ssl.

Now start the Apache server and ensure the service persists through reboots.

Next, we need to install PHP along with the necessary PHP modules. First, install yum-utils and enable the Remi PHP repo. In this instance, we’re opting to use PHP 7.1.

Lastly, we’ll install the database. We’re using MariaDB, a open-source and actively-developed fork of MySQL. Install MariaDB as follows:

Now start the MariaDB server and ensure the service persists through reboots.

Next, complete the secure installation for MariaDB.

Finally, create the Zabbix Database with the following mysql commands:

Installing Zabbix

Now that the prerequisites are satisfied, we can proceed with installing the Zabbix application from packages. First, add the Zabbix repo. Make sure you have the URL for the correct version – we want Zabbix 4.0, as shown below.

Now install the Zabbix server and web frontend.

Next, import the initial Zabbix schema into the database using the zabbix database user and password previously created.

NOTE

The ‘zabbix’ parameter after the -p is NOT the password. This is a common misconception – however, the password would have no space after the -p option. In this case, the ‘zabbix’ parameter is specifying the database for the mysql connection to use. You will be prompted to provide the password for the zabbix database user after you enter the command.

Configure the server to connect to the database as shown here. Some of these parameters may already be set correctly in the config, while others may be commented out by default.

Finally, modify the Apache config for Zabbix as follows. Comment out the section for mod_php5.c, replacing it with a stanza for PHP 7, using the parameters below. Restart Apache after saving the config.

Starting the Zabbix Server

We’re now finally ready to start the Zabbix server for the first time.

After attempting to start the server, however, we can see that the service failed to start.



SELinux Configuration

So what’s going on here? We suspect SELinux is interfering with something out-of-the-box, so let’s do some investigation to confirm. First, install the SELinux troubleshooting tools.

Now run the setroubleshoot cli tool to search the audit log for SELinux alerts.

So we can see here just how useful the SELinux troubleshooting tools are. Not only does the utility tell us exactly what is being blocked (Zabbix Server attempting create access on the zabbix_server_alerter.sock sock_file), but it also gives us the exact command we need to resolve the issue. Not so bad, eh? Simply execute the suggested commands to allow the proper access to the zabbix_server_alerter.sock file, as shown here:

Now let’s attempt to restart Zabbix and run the setroubleshoot tool again.

Now we see a similar error as before, except this time Zabbix needs access to the zabbix_server_preprocessing.sock. Again, we can allow this access with the suggested commands.

And again, restart the Zabbix server.

Now things seem much happier.

Let’s run the setroubleshoot tool once more to ensure there are no more errors.

Now that the Zabbix server appears to be happy, be sure to set the server to start automatically.

So now that Zabbix server is running, we still need to finish the web frontend install. Before we’re able to connect to the UI, however, we’ll need to open the necessary ports in the Linux firewalld daemon. If we look at the ports currently open, we can see that neither http nor https are allowed.

Simply add the following service rules with firewall-cmd to allow http/https.

While we’re at it, let’s go ahead and add the firewall rules to allow the active and passive Zabbix agent checks:

Reload the firewalld daemon to apply the new rules.

We can now view the new rules in the allowed services and ports.

And, finally, restart Apache.

Zabbix Frontend Configuration

In a browser, connect to the frontend configuration UI at http://server/zabbix where “server” is the IP address of your Zabbix server. On the welcome screen, click “Next step” to continue.

Once all the prerequisites have been satisfactorily met, click “Next step” to continue.

Provide the database connection information, using the zabbix database user and password from above and click, “Next step” to continue.

Next, configure the Zabbix server details.

Review the installation summary for accuracy and click, “Next step” to finalize the install.

The Zabbix frontend installation should now be completed. Click, “Finish” to exit.

Log into the Zabbix Admin Console with the following credentials:

Username: Admin
Password: zabbix

Upon logging in, you’ll likely see an error indicating that Zabbix is still not properly running.

Let’s head back to see our old friend, the setroubleshoot tool, to see if SELinux is again the culprit.

Sure enough. As we can see from the log, now that things are up and running, httpd is being prevented from communicating on the Zabbix port 10051. It then clearly gives us some guidance on what we need to do to allow this behavior. Run the suggested commands as follows:

Now restart Apache and Zabbix.

After refreshing the Zabbix console, we can see that our error is now gone and the server appears to be functioning properly.

SSL Configuration

As a final step before we begin to configure our new Zabbix server, let’s generate a certificate and enable SSL. Of course, if your organization has it’s own PKI, or you purchase named or wildcard certificates for your domain, you’ll want to follow those processes rather than the one shown here.

The process detailed here will generate a self-signed certificate. Replace the information below with the relevant location, server name, domain, and IP information, as it relates to your organization. First, generate a CNF file from which to generate the certificate info.

Next, generate the certificate with OpenSSL.

Edit the Apache SSL configuration to use the newly created certificate and private key. Locate the section headed “<VirtualHost _default_:443>” and edit as follows:

Restart Apache after the SSL configuration has been saved.

You should now be able to reach your Zabbix server and login via SSL.

Conclusion

That sums up the Zabbix 4.0 LTS install on CentOS 7. You should now have a working Zabbix server and be ready to configure, add hosts, roll out agents and explore the various and sundry items that Zabbix can monitor.

Resizing the Linux Root Partition in a Gen2 Hyper-V VM

Resizing the Linux Root Partition in a Gen2 Hyper-V VM

Without a doubt, modern virtualization has changed the landscape of enterprise computing forever. Since virtual machines are abstracted away from the physical hardware, changes in compute, memory, and storage resources become mere clicks of a mouse. And, as hypervisors mature, many operations that were once thought of as out-of-band tasks, such as adding storage or even memory can now be done with little, or even zero downtime.

Hyper-V SCSI Disks and Linux

In many cases, hypervisors are backed by large storage area networks (SANs). This provides shared storage for hypervisor nodes that supports failover clustering and high availability. Additionally, it gives administrators the ability to scale the virtual environment, including the ability to easily add or expand storage on existing virtual servers. Microsoft’s Hyper-V 2012 introduced Generation 2 VMs, which extends this functionality. Among the many benefits of Gen2 VMs, was the ability to boot from a SCSI disk rather than IDE. This requires UEFI rather than a legacy BIOS, so it’s only supported among newer operating systems. Many admins I talk to think this is limited to Microsoft Server 2012 and newer, probably because of the sub-optimal phrasing in the Hyper-V VM creation UI that altogether fails to mention Linux operating systems.

The fact is, however, that many newer Linux OSes also support this ability, as shown in these tables from Microsoft.

More Disk, Please

Once you’ve built a modern Linux VM and you’re booting from synthetic SCSI disks rather than emulated IDE drives, you gain numerous advantages, not the least of which is the ability to resize the OS virtual hard disk (VHDX) on the fly. This is really handy functionality – after all, what sysadmin hasn’t had an OS drive run low on disk space at some point in their career? This is simply done from the virtual machine settings in Hyper-V Manager or Failover Cluster Manager by editing the VHDX.

Now, if you’re a Microsoft gal or guy, you already know that what comes next is pretty straightforward. Open the Disk Management MMC, rescan the disks, extend the file system, and viola, you now automagically have a bigger C:\ drive. But what about for Linux VMs? Though it might be a little less intuitive, we can still accomplish the same goal of expanding the primary OS disk with zero down time in Linux.

On-the-Fly Resizing

To demonstrate this, let’s start with a vanilla, Hyper-V Generation 2, CentOS 7.6 VM with a 10GB VHDX attached to a SCSI controller in our VM. Let’s also assume we’re using the default LVM partitioning scheme during the CentOS install. Looking at the block devices in Linux, we can see that we have a 10GB disk called sda which has three partitions – sda1, sda2 and sda3. We’re interested in sda3, since that contains our root partition, which is currently 7.8GB, as demonstrated here by the lsblk command.

Now let’s take a look at df. Here we can see an XFS filesystem on our 7.8GB partition, /dev/mapper/centos-root which is mounted on root.

Finally, let’s have a look at our LVM summary:

From this information we can see that there’s currently no room to expand our physical volume or logical volume, as the entirety of /dev/sda is consumed. In the past, with a Gen1 Hyper-V virtual machine, we would have had to shut the VM down and edit the disk, since it used an emulated IDE controller. Now that we have a Gen2 CentOS VM with a SCSI controller, however, we can simply edit the disk on the fly, expanding it to 20GB.

Once the correct virtual disk is located, select the “Expand” option.

Next, provide the size of the new disk. We’ll bump this one to 20GB.

Finally, click “Finish” to resize the disk. This process should be instant for dynamic virtual hard disks, but may take a few seconds to a several minutes for fixed virtual hard disks, depending on the size of the expansion and speed of your storage subsystem. You can then verify the new disk size by inspecting the disk.

OK, so we’ve expanded the VHDX in Hyper-V, but we haven’t done anything to make our VM’s operating system aware of the new space. As seen here with lsblk, the OS is indifferent to the expanded drive.

Taking a look at parted, we again see that our /dev/sda disk is still showing 10.7GB. We need to make the CentOS operating system aware of the new space. A reboot would certainly do this, but we want to perform this entire operation with no downtime.



Issue the following command to rescan the relevant disk – sda in our case. This tells the system to rescan the SCSI bus for changes, and will report the new space to the kernel without a restart.

Now, when we look at parted again, we’re prompted to move the GPT table to the back of the disk, since the secondary table is no longer in the proper location after the VHDX expansion. Type “Fix” to correct this, and then once again to edit the GPT to use all the available disk space. Once this is complete, we can see that /dev/sda is now recognized as 20GB, but our sda3 partition is still only 10GB.

Next, from the parted CLI, next use the resizepart command to grow the partition to the end of the disk.

Our sda3 partition is now using the maximum space available, 20.2GB. The lsblk command also now correctly reports our disk as 20GB.

But what about our LVM volumes? As suspected, our physical volumes, volume groups and logical volumes all remain unchanged.

We need to first tell our pv to expand into the available disk space on the partition. Do this with the pvresize command as follows:

Sure enough, our pv is now 18.8GB with 10.00GB free. Now we need to extend the logical volume and it’s associated filesystem into the free pv space. We can do this with a single command:

Looking at our logical volumes confirms that our root lv is now 17.80GB of the 18.80GB total, or exactly 10.0GB larger than we started with, as one would expect to see.

A final confirmation with the df command illustrates that our XFS root filesystem was also resized.

Conclusion

So there you have it. Despite some hearsay to the contrary, modern Linux OSes run just fine as Gen2 VMs on Hyper-V. Coupled with a SCSI disk controller for the OS VHDX, this yields the advantage of zero-downtime root partition resizing in Linux, though it’s admittedly a few more steps than a Windows server requires. And though Linux on Hyper-V might not seem like the most intuitive choice to some sysadmins, Hyper-V has matured significantly over the past several releases and is quite a powerful and stable platform for both Linux and Windows. And one last thing – when you run critically low on disk space on Linux, don’t forget to check those reserved blocks for a quick fix!

Silicon Power Bolt B10 USB SSD Review

Silicon Power Bolt B10 USB SSD Review

It’s no secret that the advent of solid state drives (SSDs) has irrevocably changed the technology landscape. Quite simply, SSDs eliminated what has been the largest bottleneck in computing performance for quite some time – the spinning disk. With SSDs, performance metrics on workstations such as latency, throughput and input/output operations per second (IOPS) are nearly irrelevant, as other system components such as the SATA interface become the limiting factor.

There are other attributes of SSDs that aren’t often emphasized, however, such as size, weight and portability. This is likely owing in large part to the fact that drives typically conform to a standard 2.5″ or 3.5″ form factor to maintain backwards compatibility. However, crack open a typical 2.5″ SSD and it becomes pretty obvious that SSDs could be significantly smaller than the standard dictates. And though hard disk drives (HDDs) have also packed more and more data onto their 2.5″ and 3.5″ platters, it is stated that SSD storage density surpassed that of HDDs in 2016, and is expected to continue to out-pace advances in HDD recording technologies.

Portable Drives

When I travel, I always like to have some sort of portable storage with me. And, to be frank, portable is a term that’s thrown around pretty loosely these days. My mobile phone is “portable,” yet it barely fits in my front jeans pocket, and I certainly can’t climb into my buddy’s Tundra with it in there. This, it seems, has been the unfortunate trend in portable devices these days – as performance demand increases, so does the size. Often times, I simply grab a USB thumb drive or two when I’m on the road. But when it comes to thumb drives, even the best performing of the bunch leave a little to be desired. I could carry a 2.5″ external drive in my laptop bag, I suppose, but the added bulk becomes cumbersome, especially when competing for space with a mouse, extra batteries, an assortment of cables, and the laptop’s AC adapter. What I really want is something small, light, and fast. I want something easy to carry, but not limited with respect to capacity or performance – you know, like an SSD.

Silicon Power Bolt B10

Having been a longtime fan of Silicon Power’s wallet-friendly line of SSDs for our lab server OS drives, I was delighted when Silicon Power sent me a Bolt B10 portable SSD to review. This is the first portable SSD I’ve intimately used and, spoiler alert, I won’t be going back to thumb drives when I travel any time soon. Now let’s dig into the details of this little gem.

Form Factor

Conceptually, one of the things I like most about portable SSDs is that we’re finally capable of breaking existing form factor molds. Before SSDs, 2.5″ and 3.5″ hard drives defined the size and shape of portable mass storage. This meant relatively large, clunky, and generally unattractive boxes. Since internal SSDs only conform to these form factors to remain relevant in server, desktop, and laptop applications, design can take a front row with portable SSDs. Portable SSDs tend to possess sleeker lines, smaller packages, and generally more attractive aesthetics. They don’t even have to be rectangular – some manufacturers like Silicon Power even offer round flavors of portable SSDs.

The Bolt B10 conforms to a traditional rectangular shape, though by no means is this a bad thing. The drive is understated, yet attractive, and it’s credit card sized form fits comfortably in the palm of your hand. The drive is featherweight at 25g, almost too light, though not cheap feeling, and the smooth plastic just feels right. It’s the kind of combination that makes you want to fidget with it, like the satisfying click of a well-made ball point pen. You can see here that the drive has a smaller footprint than a Logitech M325 wireless mouse.

Inside the drive you’ll find the PCB that occupies about half of the actual drive enclosure volume, which is to say it’s quite small. Directing the show, you’ll find the Phison PS3111-S11 controller, which gets generally favorable reviews. We’ve had nothing but luck with Phison-controlled SSDs, so we’ve got no complaints here. You can also see the 4 x 32GB NAND chips as well as a micro-USB 3.1 Gen1 connector soldered to the PCB.

Power, Heat & Noise

One of the numerous benefits of small form-factor 2.5″ hard drives is that their enclosures can be driven solely from USB power. The Bolt B10 is no exception. It’s a USB 3.1 Gen 1 device, but is also 2.0 backwards-compatible so it’s power draw cannot exceed the USB 2.0 specification of 5 power units of 100mA, or 500mA total. At 5V this equates to a maximum power consumption 2.5 W, though I suspect the B10 draws about half of that, even when continuously reading or writing.

In fact one of the more interesting use cases (for me) of portable hard drives is slinging lossless music around when I’m on the go. Specifically, I like to be able to plug a drive into an after market stereo head-unit with front-panel USB. Unfortunately, the front-panel USB just doesn’t deliver enough power to spin up most standard 2.5″ USB drives. The B10 works flawlessly in this application, giving you up to 512 GB of easily transportable lossless music for your commute.

Additionally, solid state drives generate less heat and noise than their spinning counterparts, as one might expect. The SP Bolt B10 makes no discernible noise during operation and the tiny case feels cool to the touch even after long continuous writes.

Specifications & Features

Included in the box is the Bolt B10 and a Micro USB 3.0 cable as shown here.

Now let’s take a look at the manufacturer specifications for the B10:

Power supply DC 5V
Cable Micro-B (B10) to Type-A (PC/NB)
Capacity 128GB, 256GB, 512GB
Dimensions 80.0 x 49.5 x 9.4mm
Weight 25g
Material Plastic
Color Black
Interface USB 3.1 Gen1 / USB 3.0, USB 2.0 compatible
Performance Read(max.)400MB/s
Performance Write(max.)400MB/s
Supported OS Windows 10/8.1/8/7/Vista/XP, Mac OS 10.5.x or later, Linux 2.6.31 or later
Operating Temperature 0℃~ 70℃
Storage Temperature -40℃~ 85℃
Certification CE/FCC/BSMI/Green dot/WEEE/RoHS/EAC/KCC/RCM
Warranty 3 years

Performance and Features

Ultimately, performance is probably what people care about most in a portable SSD. The USB interface has a long history of offering underwhelming performance. USB 2.0 offered pretty measly transfer rates of 480 Mbps or 60 MB/s. Due to bus limitations, real-world speeds were closer to 35 MB/s, however. Even in 2000 when USB 2.0 was introduced, an average spinning drive could easily saturate the USB link. It wasn’t until the advent of USB 3.0, nearly 10 years later, that the USB interface was no longer the bottleneck. With transfer speeds of 5 Gbps (625 MB/s), USB 3.0 suddenly made spinning drives feel slow, and the thought of portable SSDs began to make a lot of sense.

In this case, the Bolt B10 tested was a 128 GB model and testing was performed on a modest Dell Laptop: a Latitude E6430, Core i7-3720QM CPU @ 2.60 GHz, 8 GB RAM, Silicon Power S60 120 GB SSD. Given that the Bolt B10 has theoretical maximum throughput of 400 MB/s, we should not be bottlenecked by the USB 3.0 interface.

With the queue depth set to 4, ATTO Benchmarks showed write speeds very near the claimed 400 MB/s, peaking at nearly 360 MB/s, while read speeds exceeded the listed specifications, reaching speeds of approximately 430 MB/s.

CrystalDiskMark’s numbers weren’t quite as glowing, but were still quite good overall.

Some real-world file copies yielded satisfactory results. Large file copies were generally characterized by peaking at over 280MB/s then leveling out to ~130-150MB/s for the duration of the copy.

Small file copies can be quite taxing for any storage media. The results here were also on par with other similar drives we’ve tested. Here you see the copy of the WinSXS Folder – 5.96 GB (6,405,198,876 bytes) containing 76,348 Files, 23,163 Folders.

Finally, Silicon Power lists the drive’s features as follows:

  • Ultra-compact and lightweight for great portability
  • Clean and smooth exterior design
  • Large storage capacity of up to 512GB
  • Superfast transfer rates of up to 400MB/s read & write speed*
    *The transmission speed will vary depending on system performance, such as hardware, software, file system, usage and product capacity. Speeds are tested by Silicon Power with FAT32 (for cross-platform sharing) or NTFS (for single file over 4GB) file formats using CDM or ATTO tests.
  • Supports LDPC error correction and Wear Leveling
  • Free download of SP Widget software for data backup & restore, AES 256-bit encryption, and cloud storage
  • One thing to note is that out of the box the Bolt B10 was formatted to FAT32, which is an interesting choice. As such, I could not initially copy files larger than 4 GB to the drive. Now to someone who’s been in IT for 20 years, this isn’t a big deal, and a quick reformat to NTFS resolved the issue. However, I can easily see how this might confuse someone a bit less technology savvy. Additionally, one of my pet peeves about many external hard drives are the hordes of autorun software that come pre-loaded. Most people simply want to drag and drop files to their USB drives, so this software is ordinarily just a nuisance. On a positive note, the SP Bolt B10 contains very little on the drive out of the box. In fact, the only files present on the drive were there to remind you to register your product with Silicon Power.

    Conclusion

    It should be no surprise that SSDs are now the logical choice when it comes to no-compromise portable storage. And though you’re certainly not going to tote around 4TB SSDs anytime soon (unless you have really deep pockets), affordable, portable SSDs are now large enough to meet most users’ needs. Silicon Power offers just such a drive in the Bolt B10. Are there faster portable SSDs out there? Sure, at least on paper. Considering that you’ll be tossing this in your bag and possibly leaving it on the table at the coffee shop, I’m not sure I can justify the extra cash for a few arbitrary MB/s. Additionally, it seems that many manufacturers rate their products in the lab, and under conditions that are hard to replicate in the real-world. It’s been my experience, however, that Silicon Power’s products usually meet or exceed claimed specifications. Frankly, realistic product specifications are a breath of fresh air, and make you feel like you’re getting your money’s worth, all while patronizing a company that clearly wants to earn your trust.

Joining a Windows Domain with Centrify Express

Joining a Windows Domain with Centrify Express

As you’ve seen us mention in our Linux File Servers in a Windows Domain article, Linux systems have become an omnipresent fixture of the IT landscape, even in companies that are heavily invested in Windows infrastructure. But one challenge to managing such infrastructure diversity is maintaining standardization among disparate systems. Ask any IT Admin who’s managed Linux systems over the last 20 years and you’ll likely hear numerous stories about rogue servers, shared accounts, a lack of password complexity enforcement, and a plain lack of standardization. Things have changed a lot over the past decade, however. With a general shift to virtualization, as well as provisioning tools from Ubuntu and Red Hat, and third-party tools like Ansible, system standardization on Linux has become easier than ever.

Yet another complication of a diverse environment is identity management. Especially with the ever-looming threat of virtual machine sprawl, managing disparate systems and their respective logins can be tiresome at best, and a downright security risk at worst. Thus, when considering systems and applications, a good engineer should always ask questions about identity management and integration with Active Directory, LDAP, or other directory service. Along the way Samba has offered some ability to integrate Linux systems into Active Directory, but it wasn’t always easy to implement, and identity management wasn’t Samba’s primary focus. Today, however, there are now dedicated tools to manage identities across numerous systems, including Linux, UNIX, Macs, and beyond.

One such tool that simplifies and unifies identity management across multiple platforms is Centrify. Centrify was founded in 2004 and offers software designed to thwart the number one point of entry in a data breach – compromised credentials. To say it’s a trusted platform would be an understatement – Centrify claims over half of the Fortune 100 trusts some form of their identity and access management to Centrify.

Here at Teknophiles, we look for a couple of things in a software package for use in our lab:

  1. Does the software offer Enterprise-level performance and functionality?
  2. Does the vendor provide IT professionals with an inexpensive (or free) means to test or use the software, at least in some limited capacity?

We’re not a big fan of 30-day trials at Teknophiles, because as any IT Pro knows, it’s really tough to dig into a software package and learn the ins and outs in such a limited time frame, especially with a day job. And in a lab environment, it’s almost mandatory to be able to leave a piece of hardware or software in place for an extended length of time, so that it can be tested in future configurations and scenarios (Take note, Microsoft, Re: TechNet!). We also understand that software companies are in business to make money, which is why we like the, “Free, but limited” model. In the, “Free, but limited” approach, software may be limited to a certain scope of install, number of nodes, or reduced feature set. There are lots of great examples that follow this model – Nagios Core, Thycotic Secret Server, Sophos UTM, among others, and Centrify is no exception. To get the full feature set, one must upgrade to the licensed version. Typically, there’s an easy upgrade path to the licensed version, which gives IT Admins the added confidence that they can stand up a piece of software and, if they and their superiors decide it’s a good fit, simply upgrade without standing up a completely new system.

I happen to use Centrify daily at my full-time job. It’s become instrumental in managing and standardizing access to the numerous Oracle Linux and RHEL systems we deploy. Given my experience with Centrify in the Enterprise environment, I was delighted to learn that they offer a limited package, called Centrify Express, that allows for installation on 200 servers, albeit with a reduced feature set. Here’s a table with a brief comparison of features between Centrify Infrastructure Services and Centrify Express: Reasons to Upgrade

Before You Start

In this article, we’ll briefly cover the installation of the Centrify Express Agent on CentOS 7. Before we dig in, however, there are a couple of things you need to make sure you have in order beforehand.

First, it probably goes without saying, but you need a working Active Directory environment with at least one functioning Domain Controller. You also need the credentials for a user who has the ability to add systems to the domain. It’s probably easiest if you use an account that is a Domain Admin or has similarly delegated permissions within Active Directory.

Second, as we’ve mentioned previously, DNS is a critical component of a functional Active Directory implementation. Be sure you have good FQDN (server.domain.com) resolution between the Domain Controllers and the system you wish to become a domain member, as well as name resolution from the Linux member server to the Domain Controllers (DC01.domain.com, DC02.domain.com, etc.)

Finally, give yourself at least one local Linux account with sudo access that is NOT the same as any AD account. Should you at some point find yourself unable to log into the server via Active Directory, you can use this account as a fallback.

WARNING

If the local account has the same name as an AD account, the system will assume you are trying to use the AD account and you may be unable to log in. If this is the only local account with sudo access, you could find yourself without the ability to administer the server!

With that out of the way, let’s move on to the fun parts.



Staging the Centrify Express Installation Files

Begin by downloading Centrify Express for the proper flavor of Linux here. You’ll have to fill out a short form to gain access to the installers, but it’s free and there’s no commitment. In fact, unlike some other software vendors, I haven’t been plagued by sales calls, either.

Once, you have the proper installer downloaded, simply SFTP the file to the server you wish to become a domain member. On the server, create a folder to extract the installation files.

Next, extract the installation files as follows.

Now, simply begin the installation by executing the install.sh script.

Select X for the Centrify Express installation.

Select Y to continue to install in Express mode.

Select Y to verify the AD environment. This is a good idea, as it will flag any major issues prior to attempting the domain join.

Enter the domain you wish to join.

Confirm you want to join an Active Directory Domain by selecting Y.

Again, enter the name of the domain you wish to join.

Next, enter the username and password of an admin with permission to join systems to the domain. It doesn’t have to be a Domain Admin, but it using a Domain Admin account may simplify permissions troubleshooting:

Verify the computer name and the container DN within Active Directory. In most cases the Computers container will suffice, as you can always move the machine later for organizational purposes. If you choose, you can also specify a different container, as shown here.

Now, enter the name of the Domain Controller you wish to use to join the domain. Typically this can be left as auto detect, but you can also specify a DC.

Choose whether you want the system to reboot upon completion of the install and domain join. Though this is not required, other services may need to be restarted for full integration.

Finally, confirm the options you provided and select Y to proceed.

View the output of the installation summary and resolve any issues. In this case you can see we had one warning regarding SSH configuration. Any issues may need to be addressed for complete integration.

Before logging out as root, be sure to add domain admins or other Linux administrator AD group to the sudoers file, by adding the following line.

You should now be able to log in via a domain account. Use only the user’s shortname (not FQDN).

You can quickly confirm that your AD account is working properly by viewing your user’s domain group memberships.

That’s it for the install! We hope to provide additional Centrify walk-throughs in a future article (Centrify Samba/PuTTY), but this should quickly get you up and running with single sign-on for your domain-integrated Linux servers. As you can see, Centrify provides a neat and tidy package to manage identities across multiple server platforms in a Windows Domain. It’s rock-solid, and right at home in a small home lab or large Enterprise. We’ve been using it for years now without issue in both of environments, and hope you’ll find it as useful as we have.

Network Configuration in RHEL/CentOS Linux 7

Network Configuration in RHEL/CentOS Linux 7

Although numerous and significant changes have been introduced in recent versions of Red Hat Enterprise Linux/CentOS, one of the most hotly contested topics seems to center around network configuration. In older versions of RHEL/CentOS, the default means of configuring network interfaces was via network scripts and use of the Network Administration Tool, which is now deprecated. Additionally, the well-known net-tools are also considered obsolete, replaced by the less-familiar ip commands. From RHEL/CentOS 6 onward, emphasis is now placed on using NetworkManager for network configuration, as well as the aforementioned ip command set. As of RHEL/CentOS 7, NetworkManager is considered the default method for managing network configurations, though the RHEL development teams paid careful attention to ensure that the network scrips would still function, and that they play nice with NetworkManager. Note this passage from the RHEL Networking Guide describing the relationship between NetworkManager and the older networking scripts:

“Note that in Red Hat Enterprise Linux 7, NetworkManager is started first, and /etc/init.d/network checks with NetworkManager to avoid tampering with NetworkManager’s connections. NetworkManager is intended to be the primary application using sysconfig configuration files and /etc/init.d/network is intended to be secondary, playing a fallback role.”

Old habits die hard, however, and you’ll likely see a number of articles across the Internet that detail methods to workaround NetworkManager, describe NetworkManager as, “the devil,” or “the plague,” and claim that most of the difficulties that arise with network configuration in RHEL/CentOS 7 are due to NetworkManager. While I haven’t seen any catastrophic effects from NetworkManager’s use, if it isn’t properly configured or if you aren’t well versed in its syntax, it can certainly be confusing and produce unintended results. In this article we’ll attempt to reserve judgement, and walk you through both methods of network configuration and let you decide which you prefer. First, let’s begin with NetworkManager and some of the tools used for configuration.

NetworkManager

NetworkManager is described as a “dynamic network control and configuration daemon.” Though the traditional ifcfg scripts are supported, NetworkManager is intended to the be the default mechanism for network configuration in RHEL/CentOS 7. It offers fairly extensive control of a variety of network interface types, including more modern connectivity such as Wi-Fi, mobile broadband, infiniband, as well as traditional Ethernet connections. At Teknophiles, we primarily deal with server environments in which our primary concern is the latter – traditional static Ethernet configurations. As such, much of the added functionality that NetworkManager offers will be superfluous to our discussion here. More detail on NetworkManager can be found here.

nmcli

In a server environment, NetworkManager is primarily managed by two user interfaces, nmcli (command-line interface) and nmtui (text user interface). We’ll first detail nmcli, since that’s what many administrators will prefer. Additionally, we feel strongly that understanding the cli is important since it helps to understand what is being invoked behind the scenes when using the tui. In its simplest form nmcli produces basic interface information and DNS configuration:

More detail can be had by looking at a specific network device. This command is shorthand for “nmcli device show.” Note the connection name, “Wired connection 1”.

Using the connection name, we can also look at the connection detail, which produces quite a bit of output:

This is a pretty typical configuration for a RHEL/CentOS box in its default configuration. From this output we can determine a good bit of useful information which we’ll need to configure a static IP on our server. Note that these attributes all take the format “setting.property”, with the property value listed in the right-hand column. We can see here that the connection has a human-readable ID, called “connection.id,” which is arguably somewhat unwieldy:

We can also see that the interface device is eth0, the connection type is Ethernet (802-3-ethernet), and this configuration was set by DHCP (auto).

In a typical server configuration, you’ll likely want to change several of these parameters to match your environment or standard practices. We can use the nmcli command to bring up an interactive command line tool to edit an existing connection.

The “?”, “help”, or “h” command will bring up a quick reference of available commands.

First, let’s create a more friendly connection name. Since we know this is a “connection” setting, type “goto connection” to enter the connection setting menu. Typing “print” or “p” will display the current values of this setting. The print command is contextual – it will display the values for your current location, in this case the connection menu.

We can now change connection.id value with the “set” command. Since the commands are contextual, we simply need to invoke the command as shown below. If we were not in the connection sub-menu, we would have to use the full setting.property format, connection.id, in our command.

We can see that our connection.id is now static-eth0, which is much easier to type or call in scripts, since it now lacks spaces and is much less generic. Before we continue, however, there’s one more important step. If you’ve ever configured a Cisco or other enterprise switch, you know that simply updating a config and walking away can be disastrous if you do not commit the running configuration. NetworkManager’s interactive configuration works much the same way – though you’ll see the changes to your interfaces immediately, those settings will not survive a restart unless you save the configuration. This is done with a simple “save” command.

Now let’s activate the changes and take a look at the results by looking at the connection properties again.

If you’d prefer not to use the interactive editor, this can also be accomplished via the nmcli in a single command as follows.

Let’s move on to setting the interface to use a static IPv4 address. We’ll also disable IPv6 with this command as well.

As you can see, nmcli is quite powerful. With a single command, we can very quickly modify our network configuration, without the need for modifying networks scripts or other complicated configurations.



nmtui

The nmtui utility is a text user interface designed to perform many of the same functions as nmcli. Though some Linux sysadmins distrust UIs (we’ve seen bugginess in some UIs ourselves), nmtui provides a quick way to configure a network interface via NetworkManager.

First, select the option to, “Edit a configuration,” then select, “Wired connection 1” from the list of Ethernet connections.

Next, edit the profile name, then highlight <Show> next to = IPv4 CONFIGURATION and press ENTER.

Set the following items, according to your environment:

Highlight <OK> when done and press ENTER.

Now go back to the initial NetworkManager TUI menu and select “Quit” to exit to the shell. Activate the new configuration as follows.

Finally, take a quick look at /etc/resolv.conf. You’ll notice that NetworkManager was kind enough to edit these values for you.

No School Like the Old School

As we previously stated, there’s still a great deal of loyalty to the old network scripts method for configuring networking in RHEL/CentOS 7. Since the RHEL devs made sure that network scripts still function as expected, there’s a fairly straightforward means of circumventing NetworkManager and using the network scripts. This is a boon to many sysadmins, who already have well-documented and streamlined processes in place for network configuration. To use the network scripts for configuring networking in RHEL/CentOS 7, first disable the NetworkManager.service.

NOTE

You may also elect to use the “systemctl mask” command, but this might make swapping back and forth to test each method a bit less trivial. If you’ve settled on using the network scripts for network management, we do recommend using “systemctl mask NetworkManager.service” in your final configuration.

Now that the NetworkManager daemon won’t restart upon reboot and won’t interfere with our manual configuration, we can configure our eth0 interface by creating or modifying the appropriate network script.

NOTE

Make sure this is the ONLY network script that references this device and/or MAC address. This is especially relevant if NetworkManager has been previously running things and you have made manual changes.

After saving the configuration file, restart the network service.

Alternatively, bounce the network interface.

net-tools vs. ip

Lastly, we’ll complete the discussion by looking at the old net-tools vs the ip command. There’s really not much to say here, as the net-tools have been deprecated and haven’t been updated in ages. Yet I’m sure there are those to will cling to the old tools for quite some time – I certainly find myself still relying on ifconfig and route quite a bit. On the surface each tool offers much of the same information, though truth be told, to my over-40 eyes, the old tools do provide more pleasing output.

Conclusion

As with most things Linux, there’s more than one way to skin the proverbial cat. Network configuration is no exception. Sometimes it’s difficult to let go of old methodologies and embrace the new, especially when you’ve been doing something the same way for many years. I think this is probably the case with some of the bad press surrounding NetworkManager, especially in server environments. On the other hand, older, simpler methods are sometimes faster and less error-prone than newer, less-familiar ones. In the end, it’s up to you decide which method you prefer, but both network scripts and NetworkManager should provide you the tools you need to successfully manage your Linux network configurations.

Disk Pooling in Linux with mergerFS

Disk Pooling in Linux with mergerFS

Remember the old days when we used to marvel at disk drives that measured in the hundreds of megabytes? In retrospect it seems funny now, but it wasn’t uncommon to hear someone mutter, “Man, we’ll never fill that thing up.”

If you don’t remember that, you probably don’t recall life before iPhones either, but we old timers can assure you that, once upon a time, hard drives were a mere fraction of the size they are today. Oddly enough, though, you’ll still hear the same old tripe, even from fellow IT folks. The difference now, however, is that they’re holding a helium-filled 10TB drive in their hands. But just like yesteryear, we’ll fill it up, and it’ll happen faster than you think.

In the quest to build a more scalable home IT lab, we grew tired of the old paradigm of building bigger and bigger file servers and migrating hordes of data as we outgrew the drive arrays. Rather than labor with year-over-year upgrades, we ultimately arrived at a storage solution that we feel is the best compromise of scalability, performance, and cost, while delivering SAN-NAS hybrid flexibility.

We’ll detail the full storage build in another article, but for now we want to focus specifically on the Linux filesystem that we chose for the NAS component of our storage server: mergerFS.

What is mergerFS?

Before we get into the specifics regarding mergerFS, let’s provide some relevant definitions:

Filesystem – Simply put, a filesystem is a system of organization used to store and retrieve data in a computer system. The filesystem manages the space used, filenames, directories, and file metadata such as allocated blocks, timestamps, ACLs, etc.

Union Mount – A union mount is a method for joining multiple directories into a single directory. To a user interfacing with the union, the directory would appear to contain the aggregate contents of the directories that have been combined.

FUSE – Filesystem in Userspace. FUSE is software that allows non-privileged users to create and mount their own filesystems that run as an unprivileged process or daemon. Userspace filesystems have a number of advantages. Unlike kernel-space filesystems, which are rigorously tested and reviewed prior to acceptance into the Linux Kernel, userspace filesystems, since they are abstracted away from the kernel, can be developed much more nimbly. It’s also unlikely that any filesystem that did not address mainstream requirements would be accepted into the kernel, so userspace filesystems are able to address more niche needs. We could go on about FUSE, but suffice it to say that userspace filesystems provides unique opportunities for developers to do some cool things with filesystems that were not easily attainable before.

So what is mergerFS? MergerFS is a union filesystem, similar to mhddfs, UnionFS, and aufs. MergerFS enables the joining of multiple directories that appear to the user as a single directory. This merged directory will contain all of the files and directories present in each of the joined directories. Furthermore, the merged directory will be mounted on a single point in the filesystem, greatly simplifying access and management of files and subdirectories. And, when each of the merged directories themselves are mount points representing individual disks or disk arrays, mergerFS effectively serves as a disk pooling utility, allowing us to group disparate hard disks, arrays or any combination of the two. At Teknophiles, we’ve used several union filesystems in the past, and mergerFS stands out to us for two reasons: 1) It’s extremely easy to install, configure, and use, and 2) It just plain works.

An example of two merged directories, /mnt/mergerFS01 and /mnt/mergerFS02, and the resultant union.

mergerFS vs. LVM

You might be asking yourself, “why not just use LVM, instead of mergerFS?” Though the Logical Volume Manager can address some of the same challenges that mergerFS does, such as creating a single large logical volume from numerous physical devices (disks and/or arrays), they’re quite different animals. LVM sits just above the physical disks and partitions, while mergerFS sits on top of the filesystems of those partitions. For our application, however, LVM has one significant drawback: When multiple physical volumes make up a single volume group and a logical volume is carved out from that volume group, it’s possible, and even probable, that a file will be comprised of physical extents from multiple devices. Consider the diagram below.

The practical concern here is that should an underlying physical volume in LVM fail catastrophically, be it a single disk or even a whole RAID array, any data on, or that spans, the failed volume will be impacted. Of course, the same is true of mergerFS, but since the data does not span physical volumes, it’s much easier to determine which files are impacted. There’s unfortunately no easy way that we’ve found to determine which files are located on which physical volumes in LVM.



Flexibility & Scalability

As we’ve alluded to a few times, mergerFS doesn’t care if the underlying physical volumes are single disks or RAID arrays, or LVM volumes. Since the two technologies operate at different levels with respect to the underlying disks, nothing prevents the use of LVM with mergerFS. In fact, we commonly use the following formula to create large mergerFS disk pools: multiple mdadm 4-Disk RAID10 arrays > LVM > mergerFS. In our case, the LVM is usually just used for management, we typically do not span multiple physical volumes with any volume groups, though you easily could.

This gives you incredible flexibility and scalability – want to add an individual 5400 RPM 4TB disk to an existing 12TB RAID6 array comprised of 6+2 7200 RPM 2TB drives, for 16TB total? No problem. Want to later add an LVM logical volume that spans two 8TB 2+2 RAID10 arrays for another 16TB. MergerFS is fine with that, too. In fact, mergerFS is completely agnostic to disk controller, drive size, speed, form factor, etc. With mergerFS, you can grow your storage server as you see fit.

mergerFS & Your Data

One interesting note about mergerFS is that since it is just a proxy for your data, it does not manipulate the data in any way. Prior to being part of a mergerFS union, each of your component disks, arrays, and logical volumes will already have a filesystem. This makes data recovery quite simple – should a mergerFS pool completely crash (though, unlikely), just remove the component storage devices, drop them a compatible system, mount as usual, and access your data.

What’s more, you can equally as easily add a disk to mergerFS that already has data on it. This allows you to decide at some later point if you wish to add an in-use disk to the mergerFS pool (try that with LVM). The existing data will simply show up in the mergerFS filesystem, along with whatever data is on the other volumes. It just doesn’t get any more straightforward!

mergerFS & Samba

As we stated earlier, we selected mergerFS for the NAS component of our Teknophiles Ultimate Home IT Lab storage solution. Since this is a NAS that is expected to serve files to users in a Windows Domain, we also run Samba to facilitate the Windows file sharing. Apparently, there are rumblings regarding issues with mergerFS and Samba, however, according to the author of mergerFS, this is likely due to improper Samba configuration.

Here at Teknophiles, we can unequivocally say that in server configurations based on our, “Linux File Servers in a Windows Domain,” article, Samba is perfectly stable with mergerFS. In fact, in one mergerFS pool, we’re serving up over 20TB of data spread over multiple mdadm RAID arrays. The server in question is currently sitting at 400 days of uptime, without so much as a hiccup from Samba or mergerFS.

Installing mergerFS

OK, so that’s enough background, now let’s get to the fun part. To install mergerFS, first download the latest release for your platform. We’re installing this on an Ubuntu 14.04 LTS server, so we’ll download the Trusty 64-bit .deb file. Once downloaded, install via the Debian package manager.

Creating mergerFS volumes

Now we’re going to add a couple of volumes to a mergerFS pool. You can see here that we have a virtual machine with two 5GB virtual disks, /dev/sdb and /dev/sdc. You can also see that each disk has an ext4 partition.

Next, create the mount points for these two disks and mount the disks in their respective directories.

Now we need to create a mount point for the union filesystem, which we’ll call ‘virt’ for our virtual directory.

And finally, we can mount the union filesystem. The command follows the syntax below, where <srcmounts> is a colon delimited list of directories you wish to merge.

mergerfs -o<options> <srcmounts> <mountpoint>

Additionally, you can also use globbing for the source paths, but you must escape the wildcard character.

There are numerous additional options that are available for mergerFS, but the above command will work well in most scenarios. From the mergerFS man page, here’s what the above options do:

defaults: a shortcut for FUSE’s atomic_o_trunc, auto_cache, big_writes, default_permissions, splice_move, splice_read, and splice_write. These options seem to provide the best performance.

allow_other: a libfuse option which allows users besides the one which ran mergerfs to see the filesystem. This is required for most use-cases.

use_ino: causes mergerfs to supply file/directory inodes rather than libfuse. While not a default it is generally recommended it be enabled so that hard linked files share the same inode value.

fsname=name: sets the name of the filesystem as seen in mount, df, etc. Defaults to a list of the source paths concatenated together with the longest common prefix removed.

You can now see that we have a new volume called, “mergerFS,” which is the aggregate 10GB, mounted on /mnt/virt. This new mount point can be written to, used in local applications, or served up via Samba just as any other mount point.

Other Options

Although getting a little into the weeds, it’s worth touching on an additional option in mergerFS that is both interesting and quite useful. The FUSE function policies determine how a number of different commands behave when acting upon the data in the mergerFS disk pool.

func.<func>=<policy>: sets the specific FUSE function’s policy. See below for the list of value types. Example: func.getattr=newest

Below we can see the FUSE functions and their category classifications, as well as the default policies.

Category FUSE Functions
action chmod, chown, link, removexattr, rename, rmdir, setxattr, truncate, unlink, utimens
create create, mkdir, mknod, symlink
search access, getattr, getxattr, ioctl, listxattr, open, readlink
N/A fallocate, fgetattr, fsync, ftruncate, ioctl, read, readdir, release, statfs, write
Category Default Policy
action all
create epmfs
search ff

To illustrate this a bit better, let’s look at an example. First, let’s consider file or directory creation. File and directory creation fall under the “create” category. Looking at the default policy for the create category we see that it is called, “epmfs.” From the man pages, the epmfs policy is defined as follows:

epmfs (existing path, most free space)
Of all the drives on which the relative path exists choose the drive with the most free space. For create category functions it will exclude readonly drives and those with free space less than min-freespace. Falls back to mfs.

Breaking this down further, we can see that epmfs is a “path-preserving policy,” meaning that only drives that have the existing path will be considered. This gives you a bit of control over where certain files are placed. Consider, for instance if you have four drives in your mergerFS pool, but only two of the drives contain a directory called /pictures. When using the epmfs policy, only the two drives with the pictures directory will be considered when you copy new images to the mergerFS pool.

Additionally, the epmfs policy will also serve to fill the drive with the most free space first. Once drives reach equal or near-equal capacities, epmfs will effectively round-robin the drives, as long as they also meet the existing path requirement.

There are a number of equally interesting policies, including ones that do not preserve paths (create directories as needed), fill the drive with the least free space (i.e. fill drive1, then drive2, etc.), or simply use the first drive found. Though the defaults will generally suffice, it’s a good idea to become familiar with these policies to ensure that your mergerFS configuration best suits your needs.



Adding Volumes

Similar to creating a mergerFS pool, adding disks is also quite simple. Let’s say, for instance, you want to also use the space on your existing operating system disk for the mergerFS pool. We can simply create another directory in the root filesystem to use in the union. We created ours in /mnt for demonstration purposes, but your home directory might equally suit.

Next, we unmount our existing mergerFS pool and remount it including the new directory.

Notice now our total mergerFS volume space is 19GB – the original 10GB from our two 5GB disks, plus the 9GB from /dev/sda2. And now, since we now have truly disparate volumes, let’s test our default epmfs policy. Start by creating three files in our mergerFS pool mount:

Based on our expectations of how epmfs works, we should see these files in the /mnt/mergerFS00 folder, since /dev/sda1 has the most free space.

Sure enough, this appears to work as we anticipated. Now let’s create a folder on one disk only.

Replicating our previous experiment, we’ll create a few more files, but this time in the pics directory in our mergerFS pool.

Since the epmfs policy should preserve the path pics/, and that path only exists on /mnt/mergerFS01, this is where we expect to see those files.

Removing Volumes

Removing a volume from the mergerFS pool follows the same procedure as adding a drive. Simply remount the pool without the path you wish to remove.

Notice now file1, file2, and file2 are no longer present, since they were located on /dev/sda2, which has been removed. Additionally, our total space is now back to its previous size.

mergerFS & fstab

Typically, you’re going to want your mergerFS pool to be persistent upon reboot. To do this we can simply leverage fstab as we would for any mount point. Using our example above, fstab should follow the following format:

Performance Considerations

We would be remiss to end this article without disucssing the possible performance implications of mergerFS. As with any disk pooling utility, a possible weakness of this type of this configuration is a lack of striping across drives. In RAID or LVM configurations, striping may be used to take advantage of simultaneous I/O and throughput of the available spindles. RAID level, array size, and LVM configuration play dramatically into exactly what this looks like, but even a 6+2 RAID6 array with commodity drives can achieve read speeds that will saturate a 10Gbps network. Use LVM to stripe across multiple arrays, and you can achieve stunning performance. If you’re only using single disks in your mergerFS pool, however, you’ll always be limited to the performance of a single drive. And maybe that’s OK, depending on your storage goals. Of course, careful planning of the disk subsystem and using disk arrays in your mergerFS pool can give you the best of both worlds – excellent performance and the flexibility and scalability of disk pooling.

Lastly, it’s worth noting that mergerFS is yet another layer on top of your filesystems and FUSE filesystems by their very nature add some overhead. This overhead is generally negligible, however, especially in low to moderate I/O environments. You might not want to put your busy MySQL database in userspace, but you’ll likely not notice the difference in storing your family picture albums there.

Reclaim Linux Filesystem Reserved Space

Reclaim Linux Filesystem Reserved Space

As IT Pros, we have a myriad of tools available to us to configure and tweak and tune the systems we manage. So much so, there are often everyday tools right under our noses that might have applications we may not immediately realize. In a Linux environment, tune2fs is an indispensable tool, used to tune parameters on ext2/ext3/ext4 filesystems. Most Linux sysadmins that have used mdadm software RAID will certainly recognize this utility if they’ve ever had to manipulate the stride size or stripe width in an array.

tune2fs

First, Let’s take a look at the disks on an Ubuntu file server so we can see what this tool does.

Now, we can use the tune2fs with the -l option to list the existing parameters of the filesystem superblock on /dev/sdb1.

Reserved Blocks?

As you can see, there are a number of parameters from the filesystem that we can view, including a number that can be tuned with tune2fs. In this article however, we’re going to focus on a rather simple and somewhat innocuous parameter – reserved block count. Let’s take a look at that parameter again:

At first glance, it isn’t obvious what this parameter means. In fact, I’ve worked with Linux sysadmins with years of experience that weren’t aware of this little gem. To understand this parameter, we probably have to put it’s origins in a bit of context. Once upon a time, SSDs didn’t exist, and no one knew what a terabyte was. In fact, I remember shelling out well north of a $100 for my first 20GB drive. To date myself even further, I remember the first 486-DX PC I built with my father in the early ’90s, and it’s drive was measured in megabytes. Crazy, I know. Since drive space wasn’t always so plentiful, and the consequences of running out of disk space on the root partition in a Linux system are numerous, early filesystem developers did something smart – they reserved a percentage of filesystem blocks for privileged processes. This ensured that even if disk space ran precariously low, the root user could still log in, and the system could still execute critical processes.

That magic number? Five percent.

And while five percent of that 20GB drive back in 1998 wasn’t very much space, imagine that new 4-disk RAID1/0 array you just created with 10TB WD Red Pros. That’s five percent of 20TB of usable space, or a full terabyte. You see, though this was likely intended for the root filesystem, by default this setting applies to every filesystem created. Now, I don’t know about you, but at $450 for a 10TB WD Red Pro, that’s not exactly space I’d want to throw away.



We Don’t Need No Stinking Reserved Blocks!

The good news, however, is that space isn’t lost forever. If you forget to initially set this parameter when you create the filesystem, tune2fs allows you to retroactively reclaim that space with the -m option.

Here you can see we’ve set the reserved blocks on /dev/sdb1 to 0%. Again, this isn’t something you’d want to do on a root filesystem, but for our “multimedia” drive, this is fine – more on that later. Now, let’s look at our filesystem parameters once again.

Notice now that our reserved blocks is set to zero. Finally, let’s have a look at our free disk space to see the real world impact. Initially, we had 50GB of 382GB free. Now we can see that, although neither the size of the disk nor the amount of used space has changed, we now have 69GB free, reclaiming 19GB of space.

Defrag Implications

Lastly, I’d be remiss if I didn’t mention that there’s one other function these reserved blocks serve. As always, in life there’s no such thing as a free lunch (or free space, in this case). The filesystem reserved blocks also serve to provide the system with free blocks with which to defragment the filesystem. Clearly, this isn’t something you’d want to do on a filesystem that contained a database, or in some other situation in which you had a large number of writes and deletions. However, if like in our case, you’re dealing with mostly static data, in a write once, read many (WORM) type configuration, this shouldn’t have a noticeable impact. In fact, the primary developer for tune2fs, Google’s Theodore Ts’o, can be seen here confirming this supposition.

So there you have it. You may be missing out on some valuable space, especially in those multi-terabyte arrays out there. And though it’s no longer 1998, and terabytes do come fairly cheap these days, it’s still nice to know you’re getting all that you paid for.

Silicon Power S60 60GB SSD Review

Silicon Power S60 60GB SSD Review

As we’ve mentioned in a previous article, choosing the correct hard drive for each application is critical to the performance and longevity of the drive. Different types of drives may be well-suited for some applications, but much less so for others.

Server OS Drives

One area that’s becoming somewhat challenging is finding drives to serve as operating system drives in our lab servers. On the surface, the requirements for such a drive do not seem too difficult to fulfill – we need drives that work well in a RAID1 array, display good read characteristics, are durable, and are inexpensive. Incidentally, modern consumer drives fit this bill quite well – with one caveat. You see, finding good consumer SSDs is an easy task. There are excellent options from Samsung, Crucial/Micron, Western Digital, SanDisk, Silicon Power, Toshiba/OCZ and many others. In fact, if we were looking for a 500GB SSD, the hardest part might be choosing between what seems to be an endless number of similarly performing drives.

If we were looking for a 500GB SSD, that is. In selecting a server OS drive, a 500GB drive would mostly be wasted space. A Windows Server OS might use 20GB – 30GB, unless you run a large database locally. Domain Controllers typically stay under the 15GB mark. Linux servers are considerably smaller, yet – a storage server with a few applications might consume 10GB – 15GB of disk space including swap. And it would be pretty egregious if a mail server used beyond 6GB -7GB. Obviously, this is where virtualization and shared resources becomes so advantageous. And while we firmly believe virtualization is a key component of any good home or work IT lab (which we’ll discuss in great detail later), you may not either be at the point where you need to virtualize or may not have the resources to do so. Furthermore, even if you virtualize, there are still several use cases where physical boxes are desirable or even necessary. And physical boxes need OS drives.



The Silicon Power S60

While in the hunt for a small, inexpensive, consumer SSDs, we’ve run across a few models that have worked well. Initially, we employed several of the Mushkin ECO2 60GB SSDs, as they were inexpensive, sized right, and, though they never got dreamy reviews, seemed pretty solid. In the end, however, these drives have appeared to be phased out by the manufacturer, and we have admittedly had about a 20% failure rate over a few years. So, in an effort to find a suitable replacement to the now defunct Mushkin ECO2s, we stumbled upon the Silicon Power line of SSDs. Like many SSD manufacturers, Silicon Power offers a number of SSD model ranges, and it can be sometimes hard to discern significant differences among the models. From entry-level consumer “laptop upgrade” S55 and S60 models, to “prosumer” gaming models like the S85 with a five-year warranty, as well as the TLC 3D NAND-based A55 model, Silicon Power has an offering for most applications. This ultimately led us to the Silicon Power S60. The S60 is a “consumer plus grade” SSD, which means it’s designed to fit the inexpensive laptop upgrade niche. And while it’s quite suitable for such duties, we like the fact that it’s available in a 60GB model – perfect for operating system drives. Best yet, it can be regularly found for under $35 per drive, making it a relative bargain given it’s small capacity. Consumer reviews of the S60 drive are solid as well, with nearly 200 reviews averaging 4.3 stars on Amazon at the time of this post. Since, we run a double-digit number of these drives, I always keep a couple cold spares on hand, but it’s still nice to know I can have a replacement at my door in two days if necessary (shameless Amazon Prime plug).

From the Silicon Power site, the S60’s specifications are listed as follows:

Capacity 32GB, 60GB, 120GB, 240GB, 480GB
Dimensions 100.0 X 69.9 X 7.0mm
Weight 63g (max.)
Interface SATA III
Performance Read(max.) ATTO:
480GB: 560MB/s
240GB: 550MB/s
120GB: 520MB/s
CDM:
480GB: 520MB/s
240GB: 500MB/s
120GB: 420MB/s
Performance Write(max.) ATTO:
480GB, 240GB: 500MB/s
120GB: 490MB/s
CDM:
480GB: 460MB/s
240GB: 300MB/s
120GB: 170MB/s
MTBF 1,500,000 hours
Operation Voltage 5V
Vibration Resistance Test 20G
Shock Resistance Test 1500G Max
Warranty 3 years
Note Performance result may vary, depending on system platform, software, interface and capacity.

Performance and Features

Though Silicon Power doesn’t list the performance specifications of the 60GB model, we tested the drive performance with both the manufacturer’s software, SP Toolbox, as well as ATTO Disk Benchmark and Crystal Disk Mark. These tests were performed in Windows Server 2012 R2 on a Supermicro A1SRI-2558F’s SoC SATA3 (6Gbps) ports.

As you can see, we found that this drive performs quite well for a smaller drive, and our read performance numbers in CDM actually exceeds the manufacturers marks for a 120GB drive by ten percent at 472MB/s. ATTO read numbers fell in line with what we’d expect – just shy of the numbers cited for the 120GB model at around 475MB/s.  Likewise, the write performance numbers for ATTO fell inline with expectations, though somewhat lower than 120GB model at around 265MB/s, while we exceed the listed speeds for the 120GB model in CDM by over 50% at 262MB/s!

Additionally, one might notice that the SSD controller is conspicuously absent from the drive specifications. There has previously been some concern that drive manufacturers may use multiple controllers in budget drives such as the S60. It would seem that this is indeed the case with the S60, which has been known to contain a SandForce or Phison controller. It doesn’t seem that Silicon Power is trying to hide this fact, however, as some manufacturers have been accused of, nor do they claim that it is one controller vs. the other. In fact, one can easily see that they reference both controllers in the SP SSD Firmware Update User Manual, indicating that your SSD may contain one or the other. To be truthful, we’ve had success with both controllers, though we ran the Firmware Update just to satisfy our curiosity. Unfortunately, we were not able to get the SSD Firmware Update to recognize the S60 in multiple systems, and were therefore unable to confirm whether the drive we tested contained the SandForce or Phison controller. Again, the drive performs as expected in both RAID and single-drive configurations, so this is not a major concern to us.

Finally, Silicon Power lists the drive’s features as follows:

  • Adopts MLC NAND flash and “SLC Cache Technology” to improve overall performance
  • 15 x faster than a standard 5400 HDD*
    *Based on “out-of-box performance” using a SATA Revision 3.0 motherboard. Performance result may vary, depending on system platform, software, interface, and capacity.
  • 7mm slim design suitable for ultrabooks and ultra-slim laptops
  • Supports TRIM command and garbage collection technology
  • NCQ and RAID ready
  • ECC (error correction code) technology to guarantee reliable data transmission
  • S.M.A.R.T. monitoring system
  • Low power consumption, shock and vibration-proof, noiseless and low latency
  • Free SP ToolBox software download for disk information such as self-monitoring analysis report, extent of consumption, and SSD diagnostics

It is heartening to note that RAID is specifically listed as a feature. It’s worth noting that all of our S60s are in RAID1 arrays, so while we cannot comment on stability or peformance of the S60 in parity arrays, we have successfully tested them with numerous disk controllers such as the HP P410i, LSI/Avago SAS 92xx HBAs in IT/IR modes, Adaptec SATA HostRaid, and software RAID via numerous onboard SATA controllers. All seem to work flawlessly with the S60, though controllers like the HP P410 will not report SSD Wear Status, which is not unexpected from a consumer drive without HP firmware. In a few cases, we even replaced failed Mushkin ECO2 60GB drives with an SP S60, and now have mixed RAID1 arrays with one 60GB Mushkin and one 60GB S60. Though the S60 reports slightly larger capacity than the Mushkin, rebuilds proceeded quickly and without issue.

Conclusion

All in all, we’re quite pleased with the the Silicon Power S60. And though we’re probably not using it in a capacity that Silicon Power ever intended, it seems to fill this niche nicely. Heck, we’ve even used a few of the larger S60 models for their intended purpose: to breathe new life into old laptops to throw around in the garage as dataloggers (we’re occasional drag racers) or machines for the kids to beat on. Regardless of the application, the S60 has thus far dutifully served it’s purpose. And though it may not be as fast as the latest generation of drives out there, it’s not hard to make the case that the S60 is one a heck of a value.