Installing Zabbix 4.0 LTS on CentOS 7

Installing Zabbix 4.0 LTS on CentOS 7

When it comes to enterprise monitoring solutions, there are a myriad of options to choose from, both paid and free and open-source software (FOSS). Here at Teknophiles, we’ve used just about all of them, and we can tell you that many are good, some are even great, but none are perfect. Even at some of the higher price points, we usually find something we don’t care for. Generally speaking, paying good money for monitoring software will get you support, and perhaps some ease of configuration that open source solutions do not offer.

That said, there are numerous free and open-source monitoring options that get the job done quite well. After using Nagios for many years, we’ve recently become quite enamored with Zabbix. Though not trivial to configure, especially in large environments (what monitoring solution is?), Zabbix is stable, polished and quite extensible. In this article – the first of several in a series on installing and configuring Zabbix – we’ll walk through a typical Zabbix install on Linux, using the latest long-term release from Zabbix, so you can take it for a test drive.

Zabbix Server Sizing

To get started, we’ll first assume you have a running CentOS 7 box, either a virtual or physical machine. With respect to hardware requirements, lab/test/home environments shouldn’t require more than 1-2 CPU cores and 2-4 GB of RAM. We’re currently monitoring over 3600 items, with over 2000 triggers on a virtual server with 1 CPU core and 2GB of RAM, and the server rarely shows any significant resource utilization.

Though disk usage depends on many factors, the Zabbix database will be the primary offender when it comes to disk consumption. Once Zabbix is installed and configured, the application files will change very little in terms of size, so whether or not the database is local or remote will greatly impact disk sizing for the Zabbix server. Regardless of the database location, make sure you have enough space for the database to grow as you accumulate historical data. This will be influenced by not only the number of hosts and items monitored, but also historical and trend storage periods. As a general starting point for a lab server with a local database, a root partition of 15-20 GB should suffice. You can see here that with just over 3600 items, our lab database is chewing up approximately 600 MB/month. This will likely stabilize at some point after all data retention periods are reached, but should be a consideration when sizing the disks.

Do some due diligence and size your Zabbix environment appropriately up front, however. Though this can depend on many factors, such as whether your database is local or remote, whether you choose to use MySQL InnoDB vs. PostgreSQL, the number of hosts you wish to monitor and whether or not you choose to use Zabbix Proxies, it’s much easier to provision resources earlier in the process than later, especially on physical hardware. Also, keep in mind that in medium to large installations (500-1000 hosts), disk I/O will also start become an important factor, along with CPU and memory considerations. Plan the disk subsystem accordingly. Some general sizing recommendations from the folks at Zabbix can be found here.

SELinux

For the purposes of this article, we’re also going to assume you’re a responsible Linux sysadmin and will be installing Zabbix with SELinux in enforcing mode. We strongly recommend that you leave SELinux this way. We’re aware that SELinux can be challenging and the knee-jerk reaction by many is to disable SELinux – we’ve been tempted ourselves at times. SELinux naturally adds a few steps to the process, but it’s completely manageable with the tools available, and will ultimately leave you with a better security stance. You can check the current status of SELinux as follows:

Prerequisites

The Zabbix server installation requires several prerequisite components to function – an Apache web server, PHP, and a database server. First, we’ll install the standard Apache httpd server with mod_ssl.

Now start the Apache server and ensure the service persists through reboots.

Next, we need to install PHP along with the necessary PHP modules. First, install yum-utils and enable the Remi PHP repo. In this instance, we’re opting to use PHP 7.1.

Lastly, we’ll install the database. We’re using MariaDB, a open-source and actively-developed fork of MySQL. Install MariaDB as follows:

Now start the MariaDB server and ensure the service persists through reboots.

Next, complete the secure installation for MariaDB.

Finally, create the Zabbix Database with the following mysql commands:

Installing Zabbix

Now that the prerequisites are satisfied, we can proceed with installing the Zabbix application from packages. First, add the Zabbix repo. Make sure you have the URL for the correct version – we want Zabbix 4.0, as shown below.

Now install the Zabbix server and web frontend.

Next, import the initial Zabbix schema into the database using the zabbix database user and password previously created.

NOTE

The ‘zabbix’ parameter after the -p is NOT the password. This is a common misconception – however, the password would have no space after the -p option. In this case, the ‘zabbix’ parameter is specifying the database for the mysql connection to use. You will be prompted to provide the password for the zabbix database user after you enter the command.

Configure the server to connect to the database as shown here. Some of these parameters may already be set correctly in the config, while others may be commented out by default.

Finally, modify the Apache config for Zabbix as follows. Comment out the section for mod_php5.c, replacing it with a stanza for PHP 7, using the parameters below. Restart Apache after saving the config.

Starting the Zabbix Server

We’re now finally ready to start the Zabbix server for the first time.

After attempting to start the server, however, we can see that the service failed to start.



SELinux Configuration

So what’s going on here? We suspect SELinux is interfering with something out-of-the-box, so let’s do some investigation to confirm. First, install the SELinux troubleshooting tools.

Now run the setroubleshoot cli tool to search the audit log for SELinux alerts.

So we can see here just how useful the SELinux troubleshooting tools are. Not only does the utility tell us exactly what is being blocked (Zabbix Server attempting create access on the zabbix_server_alerter.sock sock_file), but it also gives us the exact command we need to resolve the issue. Not so bad, eh? Simply execute the suggested commands to allow the proper access to the zabbix_server_alerter.sock file, as shown here:

Now let’s attempt to restart Zabbix and run the setroubleshoot tool again.

Now we see a similar error as before, except this time Zabbix needs access to the zabbix_server_preprocessing.sock. Again, we can allow this access with the suggested commands.

And again, restart the Zabbix server.

Now things seem much happier.

Let’s run the setroubleshoot tool once more to ensure there are no more errors.

Now that the Zabbix server appears to be happy, be sure to set the server to start automatically.

So now that Zabbix server is running, we still need to finish the web frontend install. Before we’re able to connect to the UI, however, we’ll need to open the necessary ports in the Linux firewalld daemon. If we look at the ports currently open, we can see that neither http nor https are allowed.

Simply add the following service rules with firewall-cmd to allow http/https.

While we’re at it, let’s go ahead and add the firewall rules to allow the active and passive Zabbix agent checks:

Reload the firewalld daemon to apply the new rules.

We can now view the new rules in the allowed services and ports.

And, finally, restart Apache.

Zabbix Frontend Configuration

In a browser, connect to the frontend configuration UI at http://server/zabbix where “server” is the IP address of your Zabbix server. On the welcome screen, click “Next step” to continue.

Once all the prerequisites have been satisfactorily met, click “Next step” to continue.

Provide the database connection information, using the zabbix database user and password from above and click, “Next step” to continue.

Next, configure the Zabbix server details.

Review the installation summary for accuracy and click, “Next step” to finalize the install.

The Zabbix frontend installation should now be completed. Click, “Finish” to exit.

Log into the Zabbix Admin Console with the following credentials:

Username: Admin
Password: zabbix

Upon logging in, you’ll likely see an error indicating that Zabbix is still not properly running.

Let’s head back to see our old friend, the setroubleshoot tool, to see if SELinux is again the culprit.

Sure enough. As we can see from the log, now that things are up and running, httpd is being prevented from communicating on the Zabbix port 10051. It then clearly gives us some guidance on what we need to do to allow this behavior. Run the suggested commands as follows:

Now restart Apache and Zabbix.

After refreshing the Zabbix console, we can see that our error is now gone and the server appears to be functioning properly.

SSL Configuration

As a final step before we begin to configure our new Zabbix server, let’s generate a certificate and enable SSL. Of course, if your organization has it’s own PKI, or you purchase named or wildcard certificates for your domain, you’ll want to follow those processes rather than the one shown here.

The process detailed here will generate a self-signed certificate. Replace the information below with the relevant location, server name, domain, and IP information, as it relates to your organization. First, generate a CNF file from which to generate the certificate info.

Next, generate the certificate with OpenSSL.

Edit the Apache SSL configuration to use the newly created certificate and private key. Locate the section headed “<VirtualHost _default_:443>” and edit as follows:

Restart Apache after the SSL configuration has been saved.

You should now be able to reach your Zabbix server and login via SSL.

Conclusion

That sums up the Zabbix 4.0 LTS install on CentOS 7. You should now have a working Zabbix server and be ready to configure, add hosts, roll out agents and explore the various and sundry items that Zabbix can monitor.

Resizing the Linux Root Partition in a Gen2 Hyper-V VM

Resizing the Linux Root Partition in a Gen2 Hyper-V VM

Without a doubt, modern virtualization has changed the landscape of enterprise computing forever. Since virtual machines are abstracted away from the physical hardware, changes in compute, memory, and storage resources become mere clicks of a mouse. And, as hypervisors mature, many operations that were once thought of as out-of-band tasks, such as adding storage or even memory can now be done with little, or even zero downtime.

Hyper-V SCSI Disks and Linux

In many cases, hypervisors are backed by large storage area networks (SANs). This provides shared storage for hypervisor nodes that supports failover clustering and high availability. Additionally, it gives administrators the ability to scale the virtual environment, including the ability to easily add or expand storage on existing virtual servers. Microsoft’s Hyper-V 2012 introduced Generation 2 VMs, which extends this functionality. Among the many benefits of Gen2 VMs, was the ability to boot from a SCSI disk rather than IDE. This requires UEFI rather than a legacy BIOS, so it’s only supported among newer operating systems. Many admins I talk to think this is limited to Microsoft Server 2012 and newer, probably because of the sub-optimal phrasing in the Hyper-V VM creation UI that altogether fails to mention Linux operating systems.

The fact is, however, that many newer Linux OSes also support this ability, as shown in these tables from Microsoft.

More Disk, Please

Once you’ve built a modern Linux VM and you’re booting from synthetic SCSI disks rather than emulated IDE drives, you gain numerous advantages, not the least of which is the ability to resize the OS virtual hard disk (VHDX) on the fly. This is really handy functionality – after all, what sysadmin hasn’t had an OS drive run low on disk space at some point in their career? This is simply done from the virtual machine settings in Hyper-V Manager or Failover Cluster Manager by editing the VHDX.

Now, if you’re a Microsoft gal or guy, you already know that what comes next is pretty straightforward. Open the Disk Management MMC, rescan the disks, extend the file system, and viola, you now automagically have a bigger C:\ drive. But what about for Linux VMs? Though it might be a little less intuitive, we can still accomplish the same goal of expanding the primary OS disk with zero down time in Linux.

On-the-Fly Resizing

To demonstrate this, let’s start with a vanilla, Hyper-V Generation 2, CentOS 7.6 VM with a 10GB VHDX attached to a SCSI controller in our VM. Let’s also assume we’re using the default LVM partitioning scheme during the CentOS install. Looking at the block devices in Linux, we can see that we have a 10GB disk called sda which has three partitions – sda1, sda2 and sda3. We’re interested in sda3, since that contains our root partition, which is currently 7.8GB, as demonstrated here by the lsblk command.

Now let’s take a look at df. Here we can see an XFS filesystem on our 7.8GB partition, /dev/mapper/centos-root which is mounted on root.

Finally, let’s have a look at our LVM summary:

From this information we can see that there’s currently no room to expand our physical volume or logical volume, as the entirety of /dev/sda is consumed. In the past, with a Gen1 Hyper-V virtual machine, we would have had to shut the VM down and edit the disk, since it used an emulated IDE controller. Now that we have a Gen2 CentOS VM with a SCSI controller, however, we can simply edit the disk on the fly, expanding it to 20GB.

Once the correct virtual disk is located, select the “Expand” option.

Next, provide the size of the new disk. We’ll bump this one to 20GB.

Finally, click “Finish” to resize the disk. This process should be instant for dynamic virtual hard disks, but may take a few seconds to a several minutes for fixed virtual hard disks, depending on the size of the expansion and speed of your storage subsystem. You can then verify the new disk size by inspecting the disk.

OK, so we’ve expanded the VHDX in Hyper-V, but we haven’t done anything to make our VM’s operating system aware of the new space. As seen here with lsblk, the OS is indifferent to the expanded drive.

Taking a look at parted, we again see that our /dev/sda disk is still showing 10.7GB. We need to make the CentOS operating system aware of the new space. A reboot would certainly do this, but we want to perform this entire operation with no downtime.



Issue the following command to rescan the relevant disk – sda in our case. This tells the system to rescan the SCSI bus for changes, and will report the new space to the kernel without a restart.

Now, when we look at parted again, we’re prompted to move the GPT table to the back of the disk, since the secondary table is no longer in the proper location after the VHDX expansion. Type “Fix” to correct this, and then once again to edit the GPT to use all the available disk space. Once this is complete, we can see that /dev/sda is now recognized as 20GB, but our sda3 partition is still only 10GB.

Next, from the parted CLI, next use the resizepart command to grow the partition to the end of the disk.

Our sda3 partition is now using the maximum space available, 20.2GB. The lsblk command also now correctly reports our disk as 20GB.

But what about our LVM volumes? As suspected, our physical volumes, volume groups and logical volumes all remain unchanged.

We need to first tell our pv to expand into the available disk space on the partition. Do this with the pvresize command as follows:

Sure enough, our pv is now 18.8GB with 10.00GB free. Now we need to extend the logical volume and it’s associated filesystem into the free pv space. We can do this with a single command:

Looking at our logical volumes confirms that our root lv is now 17.80GB of the 18.80GB total, or exactly 10.0GB larger than we started with, as one would expect to see.

A final confirmation with the df command illustrates that our XFS root filesystem was also resized.

Conclusion

So there you have it. Despite some hearsay to the contrary, modern Linux OSes run just fine as Gen2 VMs on Hyper-V. Coupled with a SCSI disk controller for the OS VHDX, this yields the advantage of zero-downtime root partition resizing in Linux, though it’s admittedly a few more steps than a Windows server requires. And though Linux on Hyper-V might not seem like the most intuitive choice to some sysadmins, Hyper-V has matured significantly over the past several releases and is quite a powerful and stable platform for both Linux and Windows. And one last thing – when you run critically low on disk space on Linux, don’t forget to check those reserved blocks for a quick fix!

Network Configuration in RHEL/CentOS Linux 7

Network Configuration in RHEL/CentOS Linux 7

Although numerous and significant changes have been introduced in recent versions of Red Hat Enterprise Linux/CentOS, one of the most hotly contested topics seems to center around network configuration. In older versions of RHEL/CentOS, the default means of configuring network interfaces was via network scripts and use of the Network Administration Tool, which is now deprecated. Additionally, the well-known net-tools are also considered obsolete, replaced by the less-familiar ip commands. From RHEL/CentOS 6 onward, emphasis is now placed on using NetworkManager for network configuration, as well as the aforementioned ip command set. As of RHEL/CentOS 7, NetworkManager is considered the default method for managing network configurations, though the RHEL development teams paid careful attention to ensure that the network scrips would still function, and that they play nice with NetworkManager. Note this passage from the RHEL Networking Guide describing the relationship between NetworkManager and the older networking scripts:

“Note that in Red Hat Enterprise Linux 7, NetworkManager is started first, and /etc/init.d/network checks with NetworkManager to avoid tampering with NetworkManager’s connections. NetworkManager is intended to be the primary application using sysconfig configuration files and /etc/init.d/network is intended to be secondary, playing a fallback role.”

Old habits die hard, however, and you’ll likely see a number of articles across the Internet that detail methods to workaround NetworkManager, describe NetworkManager as, “the devil,” or “the plague,” and claim that most of the difficulties that arise with network configuration in RHEL/CentOS 7 are due to NetworkManager. While I haven’t seen any catastrophic effects from NetworkManager’s use, if it isn’t properly configured or if you aren’t well versed in its syntax, it can certainly be confusing and produce unintended results. In this article we’ll attempt to reserve judgement, and walk you through both methods of network configuration and let you decide which you prefer. First, let’s begin with NetworkManager and some of the tools used for configuration.

NetworkManager

NetworkManager is described as a “dynamic network control and configuration daemon.” Though the traditional ifcfg scripts are supported, NetworkManager is intended to the be the default mechanism for network configuration in RHEL/CentOS 7. It offers fairly extensive control of a variety of network interface types, including more modern connectivity such as Wi-Fi, mobile broadband, infiniband, as well as traditional Ethernet connections. At Teknophiles, we primarily deal with server environments in which our primary concern is the latter – traditional static Ethernet configurations. As such, much of the added functionality that NetworkManager offers will be superfluous to our discussion here. More detail on NetworkManager can be found here.

nmcli

In a server environment, NetworkManager is primarily managed by two user interfaces, nmcli (command-line interface) and nmtui (text user interface). We’ll first detail nmcli, since that’s what many administrators will prefer. Additionally, we feel strongly that understanding the cli is important since it helps to understand what is being invoked behind the scenes when using the tui. In its simplest form nmcli produces basic interface information and DNS configuration:

More detail can be had by looking at a specific network device. This command is shorthand for “nmcli device show.” Note the connection name, “Wired connection 1”.

Using the connection name, we can also look at the connection detail, which produces quite a bit of output:

This is a pretty typical configuration for a RHEL/CentOS box in its default configuration. From this output we can determine a good bit of useful information which we’ll need to configure a static IP on our server. Note that these attributes all take the format “setting.property”, with the property value listed in the right-hand column. We can see here that the connection has a human-readable ID, called “connection.id,” which is arguably somewhat unwieldy:

We can also see that the interface device is eth0, the connection type is Ethernet (802-3-ethernet), and this configuration was set by DHCP (auto).

In a typical server configuration, you’ll likely want to change several of these parameters to match your environment or standard practices. We can use the nmcli command to bring up an interactive command line tool to edit an existing connection.

The “?”, “help”, or “h” command will bring up a quick reference of available commands.

First, let’s create a more friendly connection name. Since we know this is a “connection” setting, type “goto connection” to enter the connection setting menu. Typing “print” or “p” will display the current values of this setting. The print command is contextual – it will display the values for your current location, in this case the connection menu.

We can now change connection.id value with the “set” command. Since the commands are contextual, we simply need to invoke the command as shown below. If we were not in the connection sub-menu, we would have to use the full setting.property format, connection.id, in our command.

We can see that our connection.id is now static-eth0, which is much easier to type or call in scripts, since it now lacks spaces and is much less generic. Before we continue, however, there’s one more important step. If you’ve ever configured a Cisco or other enterprise switch, you know that simply updating a config and walking away can be disastrous if you do not commit the running configuration. NetworkManager’s interactive configuration works much the same way – though you’ll see the changes to your interfaces immediately, those settings will not survive a restart unless you save the configuration. This is done with a simple “save” command.

Now let’s activate the changes and take a look at the results by looking at the connection properties again.

If you’d prefer not to use the interactive editor, this can also be accomplished via the nmcli in a single command as follows.

Let’s move on to setting the interface to use a static IPv4 address. We’ll also disable IPv6 with this command as well.

As you can see, nmcli is quite powerful. With a single command, we can very quickly modify our network configuration, without the need for modifying networks scripts or other complicated configurations.



nmtui

The nmtui utility is a text user interface designed to perform many of the same functions as nmcli. Though some Linux sysadmins distrust UIs (we’ve seen bugginess in some UIs ourselves), nmtui provides a quick way to configure a network interface via NetworkManager.

First, select the option to, “Edit a configuration,” then select, “Wired connection 1” from the list of Ethernet connections.

Next, edit the profile name, then highlight <Show> next to = IPv4 CONFIGURATION and press ENTER.

Set the following items, according to your environment:

Highlight <OK> when done and press ENTER.

Now go back to the initial NetworkManager TUI menu and select “Quit” to exit to the shell. Activate the new configuration as follows.

Finally, take a quick look at /etc/resolv.conf. You’ll notice that NetworkManager was kind enough to edit these values for you.

No School Like the Old School

As we previously stated, there’s still a great deal of loyalty to the old network scripts method for configuring networking in RHEL/CentOS 7. Since the RHEL devs made sure that network scripts still function as expected, there’s a fairly straightforward means of circumventing NetworkManager and using the network scripts. This is a boon to many sysadmins, who already have well-documented and streamlined processes in place for network configuration. To use the network scripts for configuring networking in RHEL/CentOS 7, first disable the NetworkManager.service.

NOTE

You may also elect to use the “systemctl mask” command, but this might make swapping back and forth to test each method a bit less trivial. If you’ve settled on using the network scripts for network management, we do recommend using “systemctl mask NetworkManager.service” in your final configuration.

Now that the NetworkManager daemon won’t restart upon reboot and won’t interfere with our manual configuration, we can configure our eth0 interface by creating or modifying the appropriate network script.

NOTE

Make sure this is the ONLY network script that references this device and/or MAC address. This is especially relevant if NetworkManager has been previously running things and you have made manual changes.

After saving the configuration file, restart the network service.

Alternatively, bounce the network interface.

net-tools vs. ip

Lastly, we’ll complete the discussion by looking at the old net-tools vs the ip command. There’s really not much to say here, as the net-tools have been deprecated and haven’t been updated in ages. Yet I’m sure there are those to will cling to the old tools for quite some time – I certainly find myself still relying on ifconfig and route quite a bit. On the surface each tool offers much of the same information, though truth be told, to my over-40 eyes, the old tools do provide more pleasing output.

Conclusion

As with most things Linux, there’s more than one way to skin the proverbial cat. Network configuration is no exception. Sometimes it’s difficult to let go of old methodologies and embrace the new, especially when you’ve been doing something the same way for many years. I think this is probably the case with some of the bad press surrounding NetworkManager, especially in server environments. On the other hand, older, simpler methods are sometimes faster and less error-prone than newer, less-familiar ones. In the end, it’s up to you decide which method you prefer, but both network scripts and NetworkManager should provide you the tools you need to successfully manage your Linux network configurations.