Monit - Process and Service monitoring for Linux
Services on any server need to be up and running and do what they are supposed to do.
Automated monitoring is necessary to ensure and maximize the uptime and availability of system services on a server.
It allows system administrators to get notified and fix problems as soon as possible with minimal interruption or down time.
In this post we are going to play with a tool called Monit that can monitor services and processes on a server and send out alert emails when anything fails.
The tool itself is quite light weight and allows users to write "tests" using a very simple script style language.
Install Monit
On Debian and Ubuntu based systems Monit is available in the default repositories. Install using apt-get
$ sudo apt-get install monit
After installation monit would start as a background service and can be verified with the following command
# service monit status * monit is running
Configure Monit
The next thing to do is configure monit to monitor specific things. The configuration file for Monit is located at /etc/monit/monitrc on Ubuntu and Debian based systems.
You can either add all configurations to this file - /etc/monit/monitrc, or create additional configuration files in the /etc/monit/conf.d/ directory and monit would automatically read and load all of them
Start editing the configuration file to add the services you want to monitor.
Monit also has a log file located at /var/log/monit.log that would contain valuable information if you need to fix something
After configuration test the new configuration by running the following command
# sudo monit -t
Reload the new configuration without restarting all of monit
# sudo monit reload
Enable the web interface
The web interface allows you to check Monit reports from browser. This is useful when setting up and testing monit. Later on, its better to disable it to keep things secure.
Find the "set httpd" line in the configuration file and uncomment it. If the line doesn't exist, then just create one.
set httpd port 2812 and
# use address localhost # only accept connection from localhost
# allow localhost # allow localhost to connect to the server and
allow admin:monit # require user 'admin' with password 'monit'
# allow @monit # allow users of group 'monit' to connect (rw)
# allow @users readonly # allow users of group 'users' to connect readonly
Now access http://localhost:2812/ and the Monit web interface should open up.
Setup Email alerts
The whole idea behind monit is to get alerted whenever something unexpected occurs. Monit can automatically send out emails. Find the line "set alert" and specify the email where you want the reports to be delivered. Also specify the mailserver to use.
set alert [email protected] set mailserver localhost
Make sure that there is an smtp server (MTA) running on localhost and that it is able to deliver mails properly. Use the mail command to verify that the system is correctly configured to send out emails.
Avoid non-alert emails - Monit by default sends out emails for every thing it does or everything that happens. For example when monit starts. However we want only the alert specific emails. Modify the configuration as follows
set alert [email protected] not on {instance}
The above configuration would not email instance reports which are generated upon monit start/stop. Check monit documentation on event filters for alerts to learn more.
This is how a Monit alert mail looks like. It is raised when a disk space test runs and fails on the server.
Resource limit matched Service server_disk Date: Thu, 11 Jun 2015 15:41:43 Action: alert Host: localhost Description: space usage 61.4% matches resource limit [space usage>50.0%] Your faithful employee, Monit
Monitor services like Apache, Mysql, Postfix
Now you want to monitor your web server, database server, mail server and other daemons to ensure that services are up and running perfectly all the time. This is easy.
Just keep adding instructions in the monit configuration file to specify every service you would like to monitor and Monit would start doing it instantly.
After changing configurations, make sure to run 'sudo monit reload' to bring them in effect. While testing you can manually turn services on/off and check the monit report from the web interface.
Here are some examples -
1. HTTP service - port 80
The following example will tell Monit to check if the http server is serving on port 80 or not. If not, it would generate an alert
check host http_server with address localhost
if failed port 80 protocol http then alert
So if there is any problem with the http server(apache or nginx) an alert would be generated
Similarly you can set monitors for other ports like ftp, smtp, imap etc.
2. Monitor website status
Just monitoring port 80 is not sufficient to ensure that website is up and functioning. We need to ensure that fetching a page gives an http response code of 200. Here is the required configuration
# check that website is working proper
check host binarytides with address www.binarytides.com
if failed
port 80
protocol http
request "/"
status = 200
then alert
If either of "port 80" or "http server" or "status 200 for / url" fails then an alert would be generated. This is an effective way to check the that your site is up and available to users.
3. Apache web server - Process Monitoring
Processes are monitored using specific pid files. For example on Debian Apache web server has a pid located at /var/run/apache2/apache2.pid that can be checked to ensure that apache is running or not.
# Monitor Apache and restart if not running, also alert
check process apache with pidfile /var/run/apache2/apache2.pid
start program = "/etc/init.d/apache2 start"
stop program = "/etc/init.d/apache2 stop"
The above tells monit to monitor the process from the specified pid file. Over here, if monit does not find the process running, it would attempt to start it using the command specified in the "start program" parameter. By default, monit will also generate an alert(email) upon restarting.
Monitor only
If you do not want monit to restart apache and just inform about the event, use a different kind of configuration
# Alert if apache not running
check process apache with pidfile /run/apache2/apache2.pid
if does not exist then alert
4. Check Mysql on port 3306 - Port monitoring
In this example we monitor the mysql database not using the pid but directly by connecting to the service port which is 3306.
To check just the port number with protocol use the following configuration -
# check just port 3306
check host localmysql with address localhost
if failed port 3306 protocol mysql then alert
The following configuration shall check for mysql process and restart if not running. It would also check for port 3306 service.
check process mysql_server with pidfile /var/run/mysqld/mysqld.pid start program = "/etc/init.d/mysql start" stop program = "/etc/init.d/mysql stop" if failed host 127.0.0.1 port 3306 protocol mysql then alert
5. Monitor Postfix mail server and ports
Now lets setup a configuration to monitor postfix mail server. The configuration is identical to previous ones, just with a different pid file and port number
check host localpostfix with address localhost
if failed port 25 protocol smtp then alert
To monitor the process and restart upon failure use this -
check process postfix with pidfile /var/spool/postfix/pid/master.pid start program = "/etc/init.d/postfix start" stop program = "/etc/init.d/postfix stop" if failed port 25 protocol smtp then restart
And do not forget that while postfix is down, monit might not be able to send any emails (unless there are other mail servers specified). But monit would send them once the mail server is up again.
6. Alert repeatedly
By default Monit generates an alert only twice. Once when a test fails and other when it succeeds.
If you wish to get repeated alerts for the same failing test (if its too important for example), then use the following configuration.
set alert [email protected] with reminder on 15 cycles
The above configuration is a global setting, that sends all alerts after every 15 checks/cycles.
To repeat alerts only for specific tests, specify the alert directive within the particular test criteria like this
check filesystem datafs with path /dev/sda1 alert [email protected] with reminder on 1 cycles if space usage > 80% for 5 times within 5 cycles then alert
If you would like to be reminded fewer times but repeatedly then use something like "reminder on 10 cycles".
Check Monit status
This requires the monit http daemon to be running.
$ sudo monit status The Monit daemon 5.8.1 uptime: 0m Process 'apache' status Running monitoring status Monitored pid 1335 parent pid 1 uid 0 effective uid 0 gid 0 uptime 1h 52m children 7 memory kilobytes 29556 memory kilobytes total 241604 memory percent 0.3% memory percent total 2.9% cpu percent 0.0% cpu percent total 0.0% data collected Fri, 05 Dec 2014 11:11:18 System 'enlightened' status Running monitoring status Monitored load average [0.05] [0.26] [0.35] cpu 3.7%us 0.8%sy 0.0%wa memory usage 3613152 kB [44.2%] swap usage 0 kB [0.0%] data collected Fri, 05 Dec 2014 11:11:18
Or check the web interface at http://localhost:2812/ to check status for everything
Conclusion
The above examples show only the very simple examples of setting up monitoring tasks with Monit.
Monit is capable of doing much more complex things like pinging network hosts, running external scripts and test the output for a certain value.
Check the full documentation at -
https://mmonit.com/monit/documentation/monit.html
Configuration examples from Monit wiki page has lots of examples on how to configure monitors for various services -
http://mmonit.com/wiki/Monit/ConfigurationExamplesIf you have any feedback or questions let us know in the comments below.
