Saturday, April 20, 2013

Alert in monit

Every monitor tool should have a good alert functional to tell administrator about issues and monit is not an exception. Traditionally, monit allows us to contact with manager by sending email to report about any error happened to monitored services.

1. Enable email alert in monit

By default, if you don't define a notify statement, monit will not send alert messsages. There are two forms of alert statement:
-Global: enable alert to all services.
-Local: enable alert for individual service.
 Recipients in global and local lists are notified when service failed, recovered or changed. If you define both global and local recipients, the local recipient will override.
To set a global alert message, in global section using the "set" keyword
set alert recipient@domain.foo
When this statement is defined, all services being monitored will send messages to address recipient@domain.foo when something goes wrong.

To locally set alert function for each service
 check file test_alert with path "/home/cuong/testfile"
        alert abc@localhost
        if failed permission 0777 then alert


2.  Configurations for email notifications

2.1. Define mail server

Mail server is defined in global section using "set" keyword:
 SET MAILSERVER [hostname/ip] [port]
2.2. Alert message layout

monit provides a default layout for notification message, you can define of your own style. An example of standard layout
set mail-format {
        from:  monit@$HOST
        subject: monit alert -- $EVENT $SERVICE
        message: $EVENT Service $SERVICE
                Date: $DATE
                Action: $ACTION
                Host: $HOST
                Description: $DESCRIPTION
        monit by cuongnv at $DATE
}

Result:
From: monit@Deb6
To: cuong@localhost
Subject: monit alert -- Connection failed exim4
Date: Sat, 20 Apr 2013 14:25:12 GMT
X-Mailer: monit 5.5
MIME-Version: 1.0
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: 8bit

Connection failed Service exim4
                Date: Sat, 20 Apr 2013 21:25:12
                Action: restart
                Host: Deb6
                Description: failed protocol test [SMTP] at INET[localhost:25] via TCP -- SMTP: error receiving data -- Resource temporarily unavailable

        monit by cuongnv at Sat, 20 Apr 2013 21:25:12

Friday, April 19, 2013

Monitor your system with Monit

Previous posts covered terms and definitions about monit. This post will use those things to monitor processes running on system. Better is begin with an example of using monit in monitoring running service.

1. Example using monit to monitor sshd service

This example demonstrates configuring monit to monitor SSH service running on Debian 6 system, assuming monit is up and running well.

As mentioned in previous post, monitrc file is parted into 2 sections. One is global section with values globally applied to functions of monit. Another is spent for definitions of monitored object called service entries. Below illustration only shows the part of service:
(1)check process sshd with pidfile "/var/run/sshd.pid"
(2)        start = "/etc/init.d/ssh start"
(3)        stop = "/etc/init.d/ssh stop"
(4)        if failed port 22 protocol ssh then restart
(5)        if 5 restarts within 5 cycles then timeout
(6) alert admin@domain.foo
OK, I've just showed you about definitions of monitored service: SSH. When you're done putting these into monitrc file and restart monit, something like this will show up


2. Explanation

OK, show time is up, now it's time to explain what actually happened.

Line (1) tells monit about service to monitor, in this case is SSH. The syntax:
check process [process name] with pid [path/to/pidfile]
-check: keyword to monitor service.
-process: keyword tells monit to monitor a service.
-[process name]: unique name of process, in my example is sshd
-with pid: key word specifies the pid file.
-[path/to/pidfile]: fully path to process id of service, in my example is /var/run/sshd.pid

Line (2) (3) are defined in case you want to start or stop service by defining start and stop program.

Line (4)
if failed port 22 protocol ssh then restart
tells monit to take a test by making a connection to port 22 with ssh protocol running. And if the test statement is true (failed to connect), monit will make the action after the "then". Restart action actually executes stop first, wait up 30s for it to stop, then take start action in wait 30s for it to start. You can also specify the uid/gid runs the service.

Line (5)
if 5 restarts within 5 cycles then timeout
is another test action. The 1st 5 is the number of times restart service. Cycle is the time set in global section. If the "if" statement is true, "then" will set a timeout for service.

Line (6)
alert admin@domain.foo
will send an alert message to specified address. The definition of email server in global section will be covered later.

OK, now you can have an overview of definition for a monitored service.

3. What to monitor?

After having an overview about creating an object (process) to monitor by monit, it's time to come back to the question: which object(s) can monit monitor?

There are several objects (services) can take a "check" by monit:
-Process:
 check process [process name] with pid [path/to/pidfile]


-File:
 check file [filename] with path [path/to/file]
ex:  check file httpd.conf with path /usr/local/apache/conf/httpd.conf

-Directory:
check directory [dir_name] with path [path/to/dir]
ex:  check directory sbin with path /sbin

-Device:
check device [device_name] with path [path/to/device]
ex:  check device CDROM with path /dev/cdrom
-Remote host:
check host [name_of_hsost] with address [ip_addr_or_name]
ex: check host abc.domain.foo with address 10.1.1.1
It's possible to check the service running on remote host.

-System:
check system [system_name]
ex: check system my_host

Monit httpd authentication methods

Monit supports two types of authentication users via its tiny web server.

 1. Limit access from host and network

Monit uses access control lists to control the number of sources connect to server. It allows you to define both source by IP address or hostname. If the hostname is used, it will request the translation service to know the IP address of source.

If the host trying to connect to server but not found in access list will be logged with their IP address.
For example:
 set httpd port 2812 and
         use address 192.168.56.101
          SSL ENABLE
          PEMFILE         /opt/monit/monit.pem
          allow localhost
          allow host.abc.com
          allow 10.0.0.1
          allow 10.0.0.0/8
 With this configuration, any host NOT in the list - localhost, host.abc.com, 10.0.0.1 and network 10.0.0.0/8 will be denied from accessing to server. Log file will track the information of host trying to connect to server:
[PDT Apr  6 20:40:14] error    : monit: Denied connection from non-authorized client [192.168.56.1]
[PDT Apr  6 20:40:14] error    : monit: Denied connection from non-authorized client [192.168.56.1]
2. Basic authentication

There are several ways that server challenges clients  to send authentication information. Based on this information, server will decide act permit or deny. They could be:
-An allow statement contains a username and password separated by a colon (:) like this:

set httpd port 2812
      allow username:password
And this is CLEAR TEXT type
-Using PAM
-Using files in "htpasswd" format. If cryptographic is used, specify it in allow statement
 set httpd port 2812
       allow md5 /opt/monit/htpasswd usera userb userc

-Limit permission of user to read-only

set httpd port 2812
      allow username:password read-only

And it's possible to combine these methods to match requirements:

 set httpd port 2812 and
         use address 192.168.56.101
          SSL ENABLE
          PEMFILE         /opt/monit/monit.pem
          allow 10.0.0.0/8
          allow username:password
This configuration permits only connections from network 10.0.0.0/8 with a user "username" provides exactly pass "password".



Sunday, April 7, 2013

First steps with Monit

After sucessfully installed monit, the next step is coming close to it. You should getting familiar with some things related to monit.

1. How to start and stop monit

To start monit, you just have to type:
# monit

And to stop it:
# monit quit

2. monit configuration file

As the same with many services on Linux, monit operation is controlled by a configuration file. This file named: monitrc and have a sample file right in the extracted folder you configured. monit will try to search and use this file in priority order:
  1. ~/.monitrc
  2.  /etc/monitrc (Debian based)
  3. $sysconfdir/monitrc ( $sysconfdir is examined while configured as ./configure --sysconfdir)
  4. ./monitrc
This file is devided into 2 main parts: global part and service part.
  • Global part is place where you control everything about monit. monit will see and execute this part first to set appropriate configurations. The global set entry starts with keyword set and the item to configure. 
  • Service part is where you put your services under controlled by monit. Service entry starts with keyword check followed by  the service type to monitor.
3. Logging

An important part is monitoring monit, this should be the very first step to work with every service you attempt to control. With monit, you just have to make a set statement in monitrc file. Using this set statement:
set logfile [path/to/logfile]

4. monit httpd

monit allows us to control it via web interface with a tiny web service running at (default) port 2812. You can then determine any unused port you want. To enable this function, using a global set statement in monitrc file:
set httpd port 2812

After enabling web service for monit, you can access monit via the URL:
http://localhost:2812

You can permit monit using ssl with this tiny web service if you compiled monit with ssl supported, the steps as follow:
Generate a self-signed ceritificate: (You must have openssl installed or something providing the similar functioning. )
  • +Export environment variable for OpenSSL
    • #export OPENSSL_CONF=/etc/ssl/openssl.cnf (default path in Debian)
  • +Generate  private key and certificate
    • #openssl req -new -x509 -days 365 -nodes -out /opt/monit/monit.pem -keyout /opt/monit/monit.pem
    •  using $man openssl for more details about options
  • +Generate Diffie-Hellman parameters
    • #openssl gendh 512 >> /opt/monit/monit.pem 
  • +Change mod for .pem file to 0700
    • #chmod 0700 /opt/monit/monit.pem

After configuring self-signed certificate for monit web service, go to monitrc file and indicate monit to use SSL connection:
set httpd port 2812
    ssl enable
    pemfile /opt/monit/monit.pem

To this time, my monitrc file contains:
set httpd port 2812 and
        use address 192.168.56.101 # accept connection from remote host
        ssl enable
        pemfile         /opt/monit/monit.pem
        allow admin:monit      # require user 'admin' with password 'monit'
## Enbale logging
set logfile /var/log/monit.log
OK, bring up the browser, access to monit web service, using username/password: admin/monit

Saturday, April 6, 2013

Monit - Installation issues

Today, I started installing and using Monit   on Debian Squeeze system. Actually, this is the second time I use Debian system (the first time when I decided to switch to Debian from a long time using Fedora), so there are some issues I met when installing Monit.
  1. Repositories
    • I'm using these repos for testing server:
      • deb http://security.debian.org/ squeeze/updates main contrib
      • deb-src http://security.debian.org/ squeeze/updates main contrib
      • deb http://ftp.debian.org/debian/ squeeze-updates main contrib non-free
      • deb http://ftp.jp.debian.org/debian/ squeeze-updates main contrib non-free
      • deb-src http://ftp.jp.debian.org/debian/ squeeze-updates main contrib non-free
      • deb http://ftp.jp.debian.org/debian squeeze main
  2.  Dependencies
    • When building monit from source file, it requires you to install PAM and SSL libraries to support its functions.  If you don't want to install these functions, you can skip them from installing by specify appropriate options when configure. Executing ./configure -h for more details.
    • In this case, I want to install both PAM and SSL, the packages needed are: libssl-dev and libpam0g-dev
OK, if you can install it successfully, something like this will appear: