When it comes to enterprise monitoring solutions, there are a myriad of options to choose from, both paid and free and open-source software (FOSS). Here at Teknophiles, we’ve used just about all of them, and we can tell you that many are good, some are even great, but none are perfect. Even at some of the higher price points, we usually find something we don’t care for. Generally speaking, paying good money for monitoring software will get you support, and perhaps some ease of configuration that open source solutions do not offer.
That said, there are numerous free and open-source monitoring options that get the job done quite well. After using Nagios for many years, we’ve recently become quite enamored with Zabbix. Though not trivial to configure, especially in large environments (what monitoring solution is?), Zabbix is stable, polished and quite extensible. In this article – the first of several in a series on installing and configuring Zabbix – we’ll walk through a typical Zabbix install on Linux, using the latest long-term release from Zabbix, so you can take it for a test drive.
Zabbix Server Sizing
To get started, we’ll first assume you have a running CentOS 7 box, either a virtual or physical machine. With respect to hardware requirements, lab/test/home environments shouldn’t require more than 1-2 CPU cores and 2-4 GB of RAM. We’re currently monitoring over 3600 items, with over 2000 triggers on a virtual server with 1 CPU core and 2GB of RAM, and the server rarely shows any significant resource utilization.
Though disk usage depends on many factors, the Zabbix database will be the primary offender when it comes to disk consumption. Once Zabbix is installed and configured, the application files will change very little in terms of size, so whether or not the database is local or remote will greatly impact disk sizing for the Zabbix server. Regardless of the database location, make sure you have enough space for the database to grow as you accumulate historical data. This will be influenced by not only the number of hosts and items monitored, but also historical and trend storage periods. As a general starting point for a lab server with a local database, a root partition of 15-20 GB should suffice. You can see here that with just over 3600 items, our lab database is chewing up approximately 600 MB/month. This will likely stabilize at some point after all data retention periods are reached, but should be a consideration when sizing the disks.
Do some due diligence and size your Zabbix environment appropriately up front, however. Though this can depend on many factors, such as whether your database is local or remote, whether you choose to use MySQL InnoDB vs. PostgreSQL, the number of hosts you wish to monitor and whether or not you choose to use Zabbix Proxies, it’s much easier to provision resources earlier in the process than later, especially on physical hardware. Also, keep in mind that in medium to large installations (500-1000 hosts), disk I/O will also start become an important factor, along with CPU and memory considerations. Plan the disk subsystem accordingly. Some general sizing recommendations from the folks at Zabbix can be found here.
SELinux
For the purposes of this article, we’re also going to assume you’re a responsible Linux sysadmin and will be installing Zabbix with SELinux in enforcing mode. We strongly recommend that you leave SELinux this way. We’re aware that SELinux can be challenging and the knee-jerk reaction by many is to disable SELinux – we’ve been tempted ourselves at times. SELinux naturally adds a few steps to the process, but it’s completely manageable with the tools available, and will ultimately leave you with a better security stance. You can check the current status of SELinux as follows:
1 2 3 4 5 6 7 8 9 10 11 12 |
# sestatus SELinux status: enabled SELinuxfs mount: /sys/fs/selinux SELinux root directory: /etc/selinux Loaded policy name: targeted Current mode: enforcing Mode from config file: enforcing Policy MLS status: enabled Policy deny_unknown status: allowed Max kernel policy version: 31 |
Prerequisites
The Zabbix server installation requires several prerequisite components to function – an Apache web server, PHP, and a database server. First, we’ll install the standard Apache httpd server with mod_ssl.
1 2 3 |
# yum install -y httpd mod_ssl |
Now start the Apache server and ensure the service persists through reboots.
1 2 3 4 5 6 7 8 9 10 |
# systemctl start httpd.service # systemctl enable httpd.service # systemctl status httpd ● httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2019-01-31 11:31:48 EST; 19s ago |
Next, we need to install PHP along with the necessary PHP modules. First, install yum-utils and enable the Remi PHP repo. In this instance, we’re opting to use PHP 7.1.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
# yum install -y yum-utils # yum install -y http://rpms.remirepo.net/enterprise/remi-release-7.rpm # yum-config-manager --enable remi-php71 # yum install -y php php-gd php-bcmath php-xml php-mbstring php-ldap php-mysqli php-pecl-mysql # php --version PHP 7.1.26 (cli) (built: Jan 9 2019 08:02:33) ( NTS ) Copyright (c) 1997-2018 The PHP Group Zend Engine v3.1.0, Copyright (c) 1998-2018 Zend Technologies |
Lastly, we’ll install the database. We’re using MariaDB, a open-source and actively-developed fork of MySQL. Install MariaDB as follows:
1 2 3 |
# yum -y install mariadb-server mariadb |
Now start the MariaDB server and ensure the service persists through reboots.
1 2 3 4 5 6 7 8 9 10 |
# systemctl start mariadb.service # systemctl enable mariadb.service # systemctl status mariadb ● mariadb.service - MariaDB database server Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2019-01-31 13:17:12 EST; 7s ago |
Next, complete the secure installation for MariaDB.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
# mysql_secure_installation Enter current password for root (enter for none): Set root password? [Y/n] Y New password: Remove anonymous users? [Y/n] Y Disallow root login remotely? [Y/n] Y Remove test database and access to it? [Y/n] Y Reload privilege tables now? [Y/n] Y All done! If you've completed all of the above steps, your MariaDB installation should now be secure. Thanks for using MariaDB! |
Finally, create the Zabbix Database with the following mysql commands:
1 2 3 4 5 6 7 8 |
# mysql -uroot -p'root_db_password' MariaDB> create database zabbix character set utf8 collate utf8_bin; MariaDB> grant all privileges on zabbix.* to zabbix@localhost identified by 'zabbix_dbuser_password'; MariaDB> flush privileges; MariaDB> quit; |
Installing Zabbix
Now that the prerequisites are satisfied, we can proceed with installing the Zabbix application from packages. First, add the Zabbix repo. Make sure you have the URL for the correct version – we want Zabbix 4.0, as shown below.
1 2 3 4 5 |
# rpm -ivh https://repo.zabbix.com/zabbix/4.0/rhel/7/x86_64/zabbix-release-4.0-1.el7.noarch.rpm # yum-config-manager --enable rhel-7-server-optional-rpms |
Now install the Zabbix server and web frontend.
1 2 3 4 5 |
# yum install -y zabbix-server-mysql # yum install -y zabbix-web-mysql |
Next, import the initial Zabbix schema into the database using the zabbix database user and password previously created.
1 2 3 |
# zcat /usr/share/doc/zabbix-server-mysql*/create.sql.gz | mysql -uzabbix -p zabbix |
The ‘zabbix’ parameter after the -p is NOT the password. This is a common misconception – however, the password would have no space after the -p option. In this case, the ‘zabbix’ parameter is specifying the database for the mysql connection to use. You will be prompted to provide the password for the zabbix database user after you enter the command.
Configure the server to connect to the database as shown here. Some of these parameters may already be set correctly in the config, while others may be commented out by default.
1 2 3 4 5 6 7 8 |
# vi /etc/zabbix/zabbix_server.conf DBHost=localhost DBName=zabbix DBUser=zabbix DBPassword=zabbix_dbuser_password |
Finally, modify the Apache config for Zabbix as follows. Comment out the section for mod_php5.c, replacing it with a stanza for PHP 7, using the parameters below. Restart Apache after saving the config.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
# vi /etc/httpd/conf.d/zabbix.conf <IfModule mod_php7.c> php_value max_execution_time 300 php_value memory_limit 128M php_value post_max_size 16M php_value upload_max_filesize 2M php_value max_input_time 300 php_value max_input_vars 10000 php_value always_populate_raw_post_data -1 php_value date.timezone America/New_York </IfModule> # systemctl restart httpd |
Starting the Zabbix Server
We’re now finally ready to start the Zabbix server for the first time.
1 2 3 |
# systemctl start zabbix-server |
After attempting to start the server, however, we can see that the service failed to start.
1 2 3 4 5 6 7 |
# systemctl status zabbix-server ... Jan 31 16:20:50 localhost.localdomain systemd[1]: zabbix-server.service: control process exited, code=exited status=1 Jan 31 16:20:50 localhost.localdomain systemd[1]: Unit zabbix-server.service entered failed state. Jan 31 16:20:50 localhost.localdomain systemd[1]: zabbix-server.service failed. |
SELinux Configuration
So what’s going on here? We suspect SELinux is interfering with something out-of-the-box, so let’s do some investigation to confirm. First, install the SELinux troubleshooting tools.
1 2 3 |
# yum install -y setroubleshoot |
Now run the setroubleshoot cli tool to search the audit log for SELinux alerts.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# sealert -a /var/log/audit/audit.log found 1 alerts in /var/log/audit/audit.log -------------------------------------------------------------------------------- SELinux is preventing /usr/sbin/zabbix_server_mysql from create access on the sock_file zabbix_server_alerter.sock. ***** Plugin catchall (100. confidence) suggests ************************** If you believe that zabbix_server_mysql should be allowed create access on the zabbix_server_alerter.sock sock_file by default. Then you should report this as a bug. You can generate a local policy module to allow this access. Do allow this access for now by executing: # ausearch -c 'zabbix_server' --raw | audit2allow -M my-zabbixserver # semodule -i my-zabbixserver.pp |
So we can see here just how useful the SELinux troubleshooting tools are. Not only does the utility tell us exactly what is being blocked (Zabbix Server attempting create access on the zabbix_server_alerter.sock sock_file), but it also gives us the exact command we need to resolve the issue. Not so bad, eh? Simply execute the suggested commands to allow the proper access to the zabbix_server_alerter.sock file, as shown here:
1 2 3 4 5 |
# ausearch -c 'zabbix_server' --raw | audit2allow -M my-zabbixserver-alertersock # semodule -i my-zabbixserver-alertersock.pp |
Now let’s attempt to restart Zabbix and run the setroubleshoot tool again.
1 2 3 4 5 6 7 8 9 10 11 12 13 |
# systemctl start zabbix-server # sealert -a /var/log/audit/audit.log ... If you believe that zabbix_server_mysql should be allowed connectto access on the zabbix_server_preprocessing.sock unix_stream_socket by default. Then you should report this as a bug. You can generate a local policy module to allow this access. Do allow this access for now by executing: # ausearch -c 'zabbix_server' --raw | audit2allow -M my-zabbixserver # semodule -i my-zabbixserver.pp |
Now we see a similar error as before, except this time Zabbix needs access to the zabbix_server_preprocessing.sock. Again, we can allow this access with the suggested commands.
1 2 3 4 5 |
# ausearch -c 'zabbix_server' --raw | audit2allow -M my-zabbixserver-prepocessingsock # semodule -i my-zabbixserver-prepocessingsock.pp |
And again, restart the Zabbix server.
1 2 3 |
# systemctl restart zabbix-server |
Now things seem much happier.
1 2 3 4 5 6 7 8 9 |
# systemctl status zabbix-server ● zabbix-server.service - Zabbix Server Loaded: loaded (/usr/lib/systemd/system/zabbix-server.service; disabled; vendor preset: disabled) Active: active (running) since Thu 2019-01-31 16:40:12 EST; 6s ago Process: 17940 ExecStop=/bin/kill -SIGTERM $MAINPID (code=exited, status=0/SUCCESS) Process: 17943 ExecStart=/usr/sbin/zabbix_server -c $CONFFILE (code=exited, status=0/SUCCESS) Main PID: 17946 (zabbix_server) |
Let’s run the setroubleshoot tool once more to ensure there are no more errors.
1 2 3 4 5 |
# sealert -a /var/log/audit/audit.log 100% done found 0 alerts in /var/log/audit/audit.log |
Now that the Zabbix server appears to be happy, be sure to set the server to start automatically.
1 2 3 |
# systemctl enable zabbix-server |
So now that Zabbix server is running, we still need to finish the web frontend install. Before we’re able to connect to the UI, however, we’ll need to open the necessary ports in the Linux firewalld daemon. If we look at the ports currently open, we can see that neither http nor https are allowed.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# firewall-cmd --list-all public (active) target: default icmp-block-inversion: no interfaces: eth0 sources: services: ssh dhcpv6-client ports: protocols: masquerade: no forward-ports: source-ports: icmp-blocks: rich rules: |
Simply add the following service rules with firewall-cmd to allow http/https.
1 2 3 4 5 6 7 |
# firewall-cmd --add-service=http --permanent # firewall-cmd --add-service=https --permanent # firewall-cmd --reload |
While we’re at it, let’s go ahead and add the firewall rules to allow the active and passive Zabbix agent checks:
1 2 3 4 5 |
# firewall-cmd --permanent --add-port=10050/tcp # firewall-cmd --permanent --add-port=10051/tcp |
Reload the firewalld daemon to apply the new rules.
1 2 3 |
# firewall-cmd --reload |
We can now view the new rules in the allowed services and ports.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# firewall-cmd --list-all public (active) target: default icmp-block-inversion: no interfaces: eth0 sources: services: ssh dhcpv6-client http https ports: 10050/tcp 10051/tcp protocols: masquerade: no forward-ports: source-ports: icmp-blocks: rich rules: |
And, finally, restart Apache.
1 2 3 |
# systemctl restart httpd |
Zabbix Frontend Configuration
In a browser, connect to the frontend configuration UI at http://server/zabbix where “server” is the IP address of your Zabbix server. On the welcome screen, click “Next step” to continue.
Once all the prerequisites have been satisfactorily met, click “Next step” to continue.
Provide the database connection information, using the zabbix database user and password from above and click, “Next step” to continue.
Next, configure the Zabbix server details.
Review the installation summary for accuracy and click, “Next step” to finalize the install.
The Zabbix frontend installation should now be completed. Click, “Finish” to exit.
Log into the Zabbix Admin Console with the following credentials:
Username: Admin
Password: zabbix
Upon logging in, you’ll likely see an error indicating that Zabbix is still not properly running.
Let’s head back to see our old friend, the setroubleshoot tool, to see if SELinux is again the culprit.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
# sealert -a /var/log/audit/audit.log 100% done found 1 alerts in /var/log/audit/audit.log -------------------------------------------------------------------------------- SELinux is preventing /usr/sbin/httpd from name_connect access on the tcp_socket port 10051. ***** Plugin catchall_boolean (47.5 confidence) suggests ****************** If you want to allow httpd to can network connect Then you must tell SELinux about this by enabling the 'httpd_can_network_connect' boolean. Do setsebool -P httpd_can_network_connect 1 ***** Plugin catchall_boolean (47.5 confidence) suggests ****************** If you want to allow httpd to can connect zabbix Then you must tell SELinux about this by enabling the 'httpd_can_connect_zabbix' boolean. Do setsebool -P httpd_can_connect_zabbix 1 ***** Plugin catchall (6.38 confidence) suggests ************************** If you believe that httpd should be allowed name_connect access on the port 10051 tcp_socket by default. Then you should report this as a bug. You can generate a local policy module to allow this access. Do allow this access for now by executing: # ausearch -c 'httpd' --raw | audit2allow -M my-httpd # semodule -i my-httpd.pp |
Sure enough. As we can see from the log, now that things are up and running, httpd is being prevented from communicating on the Zabbix port 10051. It then clearly gives us some guidance on what we need to do to allow this behavior. Run the suggested commands as follows:
1 2 3 4 5 |
# setsebool -P httpd_can_network_connect 1 # setsebool -P httpd_can_connect_zabbix 1 |
Now restart Apache and Zabbix.
1 2 3 |
# systemctl restart httpd && systemctl restart zabbix-server |
After refreshing the Zabbix console, we can see that our error is now gone and the server appears to be functioning properly.
SSL Configuration
As a final step before we begin to configure our new Zabbix server, let’s generate a certificate and enable SSL. Of course, if your organization has it’s own PKI, or you purchase named or wildcard certificates for your domain, you’ll want to follow those processes rather than the one shown here.
The process detailed here will generate a self-signed certificate. Replace the information below with the relevant location, server name, domain, and IP information, as it relates to your organization. First, generate a CNF file from which to generate the certificate info.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
# vi /etc/pki/tls/zabbix-test01.cnf [req] default_bits = 2048 prompt = no default_md = sha256 x509_extensions = v3_req distinguished_name = dn [dn] C = US ST = Your State L = Your City/Town O = Your Organization OU = Your Business Unit CN = servername.domain.com [v3_req] subjectAltName = @alt_names [alt_names] DNS.1 = servername.domain.com DNS.2 = zabbix.domain.com IP.1 = x.x.x.x |
Next, generate the certificate with OpenSSL.
1 2 3 |
# openssl req -x509 -sha256 -nodes -days 730 -newkey rsa:2048 -config /etc/pki/tls/zabbix-test01.cnf -keyout /etc/pki/tls/private/zabbix-test01.key -out /etc/pki/tls/certs/zabbix-test01.crt |
Edit the Apache SSL configuration to use the newly created certificate and private key. Locate the section headed “<VirtualHost _default_:443>” and edit as follows:
1 2 3 4 5 6 7 8 |
# vi /etc/httpd/conf.d/ssl.conf DocumentRoot "/usr/share/zabbix" ServerName zabbix.domain.com:443 SSLCertificateFile /etc/pki/tls/certs/zabbix-test01.crt SSLCertificateKeyFile /etc/pki/tls/private/zabbix-test01.key |
Restart Apache after the SSL configuration has been saved.
1 2 3 |
# systemctl restart httpd |
You should now be able to reach your Zabbix server and login via SSL.
Conclusion
That sums up the Zabbix 4.0 LTS install on CentOS 7. You should now have a working Zabbix server and be ready to configure, add hosts, roll out agents and explore the various and sundry items that Zabbix can monitor.