Installing Zabbix 4.0 LTS on CentOS 7

Installing Zabbix 4.0 LTS on CentOS 7

When it comes to enterprise monitoring solutions, there are a myriad of options to choose from, both paid and free and open-source software (FOSS). Here at Teknophiles, we’ve used just about all of them, and we can tell you that many are good, some are even great, but none are perfect. Even at some of the higher price points, we usually find something we don’t care for. Generally speaking, paying good money for monitoring software will get you support, and perhaps some ease of configuration that open source solutions do not offer.

That said, there are numerous free and open-source monitoring options that get the job done quite well. After using Nagios for many years, we’ve recently become quite enamored with Zabbix. Though not trivial to configure, especially in large environments (what monitoring solution is?), Zabbix is stable, polished and quite extensible. In this article – the first of several in a series on installing and configuring Zabbix – we’ll walk through a typical Zabbix install on Linux, using the latest long-term release from Zabbix, so you can take it for a test drive.

Zabbix Server Sizing

To get started, we’ll first assume you have a running CentOS 7 box, either a virtual or physical machine. With respect to hardware requirements, lab/test/home environments shouldn’t require more than 1-2 CPU cores and 2-4 GB of RAM. We’re currently monitoring over 3600 items, with over 2000 triggers on a virtual server with 1 CPU core and 2GB of RAM, and the server rarely shows any significant resource utilization.

Though disk usage depends on many factors, the Zabbix database will be the primary offender when it comes to disk consumption. Once Zabbix is installed and configured, the application files will change very little in terms of size, so whether or not the database is local or remote will greatly impact disk sizing for the Zabbix server. Regardless of the database location, make sure you have enough space for the database to grow as you accumulate historical data. This will be influenced by not only the number of hosts and items monitored, but also historical and trend storage periods. As a general starting point for a lab server with a local database, a root partition of 15-20 GB should suffice. You can see here that with just over 3600 items, our lab database is chewing up approximately 600 MB/month. This will likely stabilize at some point after all data retention periods are reached, but should be a consideration when sizing the disks.

Do some due diligence and size your Zabbix environment appropriately up front, however. Though this can depend on many factors, such as whether your database is local or remote, whether you choose to use MySQL InnoDB vs. PostgreSQL, the number of hosts you wish to monitor and whether or not you choose to use Zabbix Proxies, it’s much easier to provision resources earlier in the process than later, especially on physical hardware. Also, keep in mind that in medium to large installations (500-1000 hosts), disk I/O will also start become an important factor, along with CPU and memory considerations. Plan the disk subsystem accordingly. Some general sizing recommendations from the folks at Zabbix can be found here.

SELinux

For the purposes of this article, we’re also going to assume you’re a responsible Linux sysadmin and will be installing Zabbix with SELinux in enforcing mode. We strongly recommend that you leave SELinux this way. We’re aware that SELinux can be challenging and the knee-jerk reaction by many is to disable SELinux – we’ve been tempted ourselves at times. SELinux naturally adds a few steps to the process, but it’s completely manageable with the tools available, and will ultimately leave you with a better security stance. You can check the current status of SELinux as follows:

Prerequisites

The Zabbix server installation requires several prerequisite components to function – an Apache web server, PHP, and a database server. First, we’ll install the standard Apache httpd server with mod_ssl.

Now start the Apache server and ensure the service persists through reboots.

Next, we need to install PHP along with the necessary PHP modules. First, install yum-utils and enable the Remi PHP repo. In this instance, we’re opting to use PHP 7.1.

Lastly, we’ll install the database. We’re using MariaDB, a open-source and actively-developed fork of MySQL. Install MariaDB as follows:

Now start the MariaDB server and ensure the service persists through reboots.

Next, complete the secure installation for MariaDB.

Finally, create the Zabbix Database with the following mysql commands:

Installing Zabbix

Now that the prerequisites are satisfied, we can proceed with installing the Zabbix application from packages. First, add the Zabbix repo. Make sure you have the URL for the correct version – we want Zabbix 4.0, as shown below.

Now install the Zabbix server and web frontend.

Next, import the initial Zabbix schema into the database using the zabbix database user and password previously created.

NOTE

The ‘zabbix’ parameter after the -p is NOT the password. This is a common misconception – however, the password would have no space after the -p option. In this case, the ‘zabbix’ parameter is specifying the database for the mysql connection to use. You will be prompted to provide the password for the zabbix database user after you enter the command.

Configure the server to connect to the database as shown here. Some of these parameters may already be set correctly in the config, while others may be commented out by default.

Finally, modify the Apache config for Zabbix as follows. Comment out the section for mod_php5.c, replacing it with a stanza for PHP 7, using the parameters below. Restart Apache after saving the config.

Starting the Zabbix Server

We’re now finally ready to start the Zabbix server for the first time.

After attempting to start the server, however, we can see that the service failed to start.



SELinux Configuration

So what’s going on here? We suspect SELinux is interfering with something out-of-the-box, so let’s do some investigation to confirm. First, install the SELinux troubleshooting tools.

Now run the setroubleshoot cli tool to search the audit log for SELinux alerts.

So we can see here just how useful the SELinux troubleshooting tools are. Not only does the utility tell us exactly what is being blocked (Zabbix Server attempting create access on the zabbix_server_alerter.sock sock_file), but it also gives us the exact command we need to resolve the issue. Not so bad, eh? Simply execute the suggested commands to allow the proper access to the zabbix_server_alerter.sock file, as shown here:

Now let’s attempt to restart Zabbix and run the setroubleshoot tool again.

Now we see a similar error as before, except this time Zabbix needs access to the zabbix_server_preprocessing.sock. Again, we can allow this access with the suggested commands.

And again, restart the Zabbix server.

Now things seem much happier.

Let’s run the setroubleshoot tool once more to ensure there are no more errors.

Now that the Zabbix server appears to be happy, be sure to set the server to start automatically.

So now that Zabbix server is running, we still need to finish the web frontend install. Before we’re able to connect to the UI, however, we’ll need to open the necessary ports in the Linux firewalld daemon. If we look at the ports currently open, we can see that neither http nor https are allowed.

Simply add the following service rules with firewall-cmd to allow http/https.

While we’re at it, let’s go ahead and add the firewall rules to allow the active and passive Zabbix agent checks:

Reload the firewalld daemon to apply the new rules.

We can now view the new rules in the allowed services and ports.

And, finally, restart Apache.

Zabbix Frontend Configuration

In a browser, connect to the frontend configuration UI at http://server/zabbix where “server” is the IP address of your Zabbix server. On the welcome screen, click “Next step” to continue.

Once all the prerequisites have been satisfactorily met, click “Next step” to continue.

Provide the database connection information, using the zabbix database user and password from above and click, “Next step” to continue.

Next, configure the Zabbix server details.

Review the installation summary for accuracy and click, “Next step” to finalize the install.

The Zabbix frontend installation should now be completed. Click, “Finish” to exit.

Log into the Zabbix Admin Console with the following credentials:

Username: Admin
Password: zabbix

Upon logging in, you’ll likely see an error indicating that Zabbix is still not properly running.

Let’s head back to see our old friend, the setroubleshoot tool, to see if SELinux is again the culprit.

Sure enough. As we can see from the log, now that things are up and running, httpd is being prevented from communicating on the Zabbix port 10051. It then clearly gives us some guidance on what we need to do to allow this behavior. Run the suggested commands as follows:

Now restart Apache and Zabbix.

After refreshing the Zabbix console, we can see that our error is now gone and the server appears to be functioning properly.

SSL Configuration

As a final step before we begin to configure our new Zabbix server, let’s generate a certificate and enable SSL. Of course, if your organization has it’s own PKI, or you purchase named or wildcard certificates for your domain, you’ll want to follow those processes rather than the one shown here.

The process detailed here will generate a self-signed certificate. Replace the information below with the relevant location, server name, domain, and IP information, as it relates to your organization. First, generate a CNF file from which to generate the certificate info.

Next, generate the certificate with OpenSSL.

Edit the Apache SSL configuration to use the newly created certificate and private key. Locate the section headed “<VirtualHost _default_:443>” and edit as follows:

Restart Apache after the SSL configuration has been saved.

You should now be able to reach your Zabbix server and login via SSL.

Conclusion

That sums up the Zabbix 4.0 LTS install on CentOS 7. You should now have a working Zabbix server and be ready to configure, add hosts, roll out agents and explore the various and sundry items that Zabbix can monitor.

Joining a Windows Domain with Centrify Express

Joining a Windows Domain with Centrify Express

As you’ve seen us mention in our Linux File Servers in a Windows Domain article, Linux systems have become an omnipresent fixture of the IT landscape, even in companies that are heavily invested in Windows infrastructure. But one challenge to managing such infrastructure diversity is maintaining standardization among disparate systems. Ask any IT Admin who’s managed Linux systems over the last 20 years and you’ll likely hear numerous stories about rogue servers, shared accounts, a lack of password complexity enforcement, and a plain lack of standardization. Things have changed a lot over the past decade, however. With a general shift to virtualization, as well as provisioning tools from Ubuntu and Red Hat, and third-party tools like Ansible, system standardization on Linux has become easier than ever.

Yet another complication of a diverse environment is identity management. Especially with the ever-looming threat of virtual machine sprawl, managing disparate systems and their respective logins can be tiresome at best, and a downright security risk at worst. Thus, when considering systems and applications, a good engineer should always ask questions about identity management and integration with Active Directory, LDAP, or other directory service. Along the way Samba has offered some ability to integrate Linux systems into Active Directory, but it wasn’t always easy to implement, and identity management wasn’t Samba’s primary focus. Today, however, there are now dedicated tools to manage identities across numerous systems, including Linux, UNIX, Macs, and beyond.

One such tool that simplifies and unifies identity management across multiple platforms is Centrify. Centrify was founded in 2004 and offers software designed to thwart the number one point of entry in a data breach – compromised credentials. To say it’s a trusted platform would be an understatement – Centrify claims over half of the Fortune 100 trusts some form of their identity and access management to Centrify.

Here at Teknophiles, we look for a couple of things in a software package for use in our lab:

  1. Does the software offer Enterprise-level performance and functionality?
  2. Does the vendor provide IT professionals with an inexpensive (or free) means to test or use the software, at least in some limited capacity?

We’re not a big fan of 30-day trials at Teknophiles, because as any IT Pro knows, it’s really tough to dig into a software package and learn the ins and outs in such a limited time frame, especially with a day job. And in a lab environment, it’s almost mandatory to be able to leave a piece of hardware or software in place for an extended length of time, so that it can be tested in future configurations and scenarios (Take note, Microsoft, Re: TechNet!). We also understand that software companies are in business to make money, which is why we like the, “Free, but limited” model. In the, “Free, but limited” approach, software may be limited to a certain scope of install, number of nodes, or reduced feature set. There are lots of great examples that follow this model – Nagios Core, Thycotic Secret Server, Sophos UTM, among others, and Centrify is no exception. To get the full feature set, one must upgrade to the licensed version. Typically, there’s an easy upgrade path to the licensed version, which gives IT Admins the added confidence that they can stand up a piece of software and, if they and their superiors decide it’s a good fit, simply upgrade without standing up a completely new system.

I happen to use Centrify daily at my full-time job. It’s become instrumental in managing and standardizing access to the numerous Oracle Linux and RHEL systems we deploy. Given my experience with Centrify in the Enterprise environment, I was delighted to learn that they offer a limited package, called Centrify Express, that allows for installation on 200 servers, albeit with a reduced feature set. Here’s a table with a brief comparison of features between Centrify Infrastructure Services and Centrify Express: Reasons to Upgrade

Before You Start

In this article, we’ll briefly cover the installation of the Centrify Express Agent on CentOS 7. Before we dig in, however, there are a couple of things you need to make sure you have in order beforehand.

First, it probably goes without saying, but you need a working Active Directory environment with at least one functioning Domain Controller. You also need the credentials for a user who has the ability to add systems to the domain. It’s probably easiest if you use an account that is a Domain Admin or has similarly delegated permissions within Active Directory.

Second, as we’ve mentioned previously, DNS is a critical component of a functional Active Directory implementation. Be sure you have good FQDN (server.domain.com) resolution between the Domain Controllers and the system you wish to become a domain member, as well as name resolution from the Linux member server to the Domain Controllers (DC01.domain.com, DC02.domain.com, etc.)

Finally, give yourself at least one local Linux account with sudo access that is NOT the same as any AD account. Should you at some point find yourself unable to log into the server via Active Directory, you can use this account as a fallback.

WARNING

If the local account has the same name as an AD account, the system will assume you are trying to use the AD account and you may be unable to log in. If this is the only local account with sudo access, you could find yourself without the ability to administer the server!

With that out of the way, let’s move on to the fun parts.



Staging the Centrify Express Installation Files

Begin by downloading Centrify Express for the proper flavor of Linux here. You’ll have to fill out a short form to gain access to the installers, but it’s free and there’s no commitment. In fact, unlike some other software vendors, I haven’t been plagued by sales calls, either.

Once, you have the proper installer downloaded, simply SFTP the file to the server you wish to become a domain member. On the server, create a folder to extract the installation files.

Next, extract the installation files as follows.

Now, simply begin the installation by executing the install.sh script.

Select X for the Centrify Express installation.

Select Y to continue to install in Express mode.

Select Y to verify the AD environment. This is a good idea, as it will flag any major issues prior to attempting the domain join.

Enter the domain you wish to join.

Confirm you want to join an Active Directory Domain by selecting Y.

Again, enter the name of the domain you wish to join.

Next, enter the username and password of an admin with permission to join systems to the domain. It doesn’t have to be a Domain Admin, but it using a Domain Admin account may simplify permissions troubleshooting:

Verify the computer name and the container DN within Active Directory. In most cases the Computers container will suffice, as you can always move the machine later for organizational purposes. If you choose, you can also specify a different container, as shown here.

Now, enter the name of the Domain Controller you wish to use to join the domain. Typically this can be left as auto detect, but you can also specify a DC.

Choose whether you want the system to reboot upon completion of the install and domain join. Though this is not required, other services may need to be restarted for full integration.

Finally, confirm the options you provided and select Y to proceed.

View the output of the installation summary and resolve any issues. In this case you can see we had one warning regarding SSH configuration. Any issues may need to be addressed for complete integration.

Before logging out as root, be sure to add domain admins or other Linux administrator AD group to the sudoers file, by adding the following line.

You should now be able to log in via a domain account. Use only the user’s shortname (not FQDN).

You can quickly confirm that your AD account is working properly by viewing your user’s domain group memberships.

That’s it for the install! We hope to provide additional Centrify walk-throughs in a future article (Centrify Samba/PuTTY), but this should quickly get you up and running with single sign-on for your domain-integrated Linux servers. As you can see, Centrify provides a neat and tidy package to manage identities across multiple server platforms in a Windows Domain. It’s rock-solid, and right at home in a small home lab or large Enterprise. We’ve been using it for years now without issue in both of environments, and hope you’ll find it as useful as we have.