Email Alerts from MO71

From time to time we are asked how to get MO71 to email a message when a particular event occurs. So, in this post, we will show the three main situations where you might want MO71 to do this for you.

The first thing we need, of course, is a method of sending an email. MO71 doesn’t have this support directly built in to it but it is capable of issuing an OS command. All we need therefore is a command line utility capable of sending an email. There are, in fact, many such utilities but our favourite, and one used by many of our customers, has to be ‘Blat’. This useful utility is available free for download from sourceforge.net.

Once you have installed ‘Blat’ we recommend that you construct a simple windows batch file to invoke it. To get you started here’s one, which we call blatit.bat, that we have used quite successfully.

@set from=mo71@acme.com
@set to=admin@acme.com
@set server=smtp.acme.com
@set subject="MO71 Alert"
@if (%1) == () goto NO_MESSAGE
blat.exe blat.bat -body %1 -to %to% -f %from% -s %subject% -server %server% -debug -log blat.log -timestamp
goto MOD_EXIT
:NO_MESSAGE
@echo Please enter a message to send
:MOD_EXIT

Clearly the batch file will need to be modified to suit your particular needs. At the very least you will need to change the ‘from’ and ‘to’ email addresses and the email server. However, if you take a look at the blat documentation you will see that there are a huge number of options available to you which may be appropriate. Here we are keeping things simple and just showing how you can use blat to send a simple message. The key thing about this batch file is that we have reduced the complexity of invoking blat down to a simple batch file which takes a single parameter. That parameter is the content of the message. The only other thing we should do is ensure that this batch file is in the windows path so that we don’t need to continually specify the fully qualified path. We would suggest you test your batch file outside of MO71 initially just to prove that everything is set up correctly. Entering a command like:-

blatit “Does this work”

should result in you receiving this simple test message in your email. If it doesn’t you need to re-look at the batch file settings. Of course once you do get it working you may wish to remove the –debug and –log parameters which are there primarily for debugging purposes.

So, we now have a way of sending an email. What we need now is for MO71 to invoke blatit.bat. As we said before, there are a number of places where you may wish MO71 to send an email. The key ones are:

  1. When MO71 is monitoring a Queue Manager and detects a change in its availability
  2. When MO71 catches and processes an MQ event message
    These events are fixed and defined by the IBM MQ development team. Conditions such as Queue Full, Queue Depth High, Channels Stopped etc are included.
  3. When MO71 is monitoring attributes of a queue manager and wishes to report a condition.
    This type of monitoring allows you to monitor more complicated and tailored conditions than the simple MQ events but it does have the disadvantage that MO71 has to periodically poll the Queue Manager for the information.

We shall show an example of each of these in turn.

Monitoring a Queue Manager

This is the simplest type of monitoring and is actually the original reason MO71 was written. This type of monitoring is merely concerned with whether a Queue Manager is ready and available and responding to requests.

In the location dialog for each location you will see there is a ‘Monitor’ tab. In this tab you can define whether MO71 should periodically monitor this location and how often it should do so. In addition you can define ‘Command OK’ and ‘Command Fail’ commands. These are commands which are issued when MO71 detects that the status of the Queue Manager has changed.

Therefore to get MO71 to email you when a Queue Manager is no longer responding it is a simple case of adding a command like the following to the ‘Command Fail’ field.

cmd blatit “Queue Manager MQ800 is not responding.”

The ‘cmd’ at the front of the command merely tells MO71 to invoke the command interpreter. The following values are then passed to the command interpreter as though they had been entered on a command line.

In a similar way you can add a ‘Command OK’ command so that you are also emailed when the Queue Manager becomes available. So, our dialog would look at little like this….

MO71 Monitor Tab

The Monitor Tab on the MO71 Location dialog

So, now with just these simple changes MO71 will now email us whenever the accessibility of our Queue Manager changes. Note that this monitoring will check that not only the Queue Manager is available but so is the route to the Queue Manager. If either one fails for any reason and email will be sent reporting the failure.

Monitoring for an event message

When IBM MQ was designed it was recognised that certain events within the MQ system were important enough to warrant drawing attention to them. MQ does this in the most appropriate way, which is to generate an MQ message. These special messages are event messages and as time goes by more and more event messages have been added to the MQ repertoire. The advantage of MQ event messages are that they are relatively cheap in terms of resource usage. The disadvantage of event messages is that despite there being quite a number of event message types there are only a very limited set of conditions which are monitored. The other major disadvantage is that no part of the IBM MQ product itself will act upon these messages. They are generated by the system but it is up to the user to catch these messages and do something with them. This is where MO71 comes in. Not only can you catch the event messages but you can choose to do a variety of things when one is caught. In this case, of course, we want to send an email.

So, let’s look at how we might send an email if we get a Queue High event. A Queue High event is an indication that a Queue is getting full and that perhaps the administrator should do something about it so it seems an appropriate event to choose.

IBM MQ has a number of queues where it will place event messages so the first thing we need to do is determine, from the MQ documentation, which queue we need to monitor to see the events we are interested in. The ‘Queue High’ event is placed on the SYSTEM.ADMIN.PERFM.EVENT queue. The next thing we need to do is to ensure that performance events are enabled. Firstly we should ensure that performance events are enabled for the Queue Manager as a whole by changing the Queue Manager attribute. Secondly you should ensure that the queues we are interested in have their ‘High’ performance event enabled.

So, now all we need to do is to tell MO71 to monitor the SYSTEM.ADMIN.PERFM.EVENT queue and respond to a ‘Queue High’ message by calling blatit.

The way to do this is fairly straightforward. From the main MO71 window select Action→Event. This will bring up a dialog which allows you to enter the various elements about the event that you want to process. Fill in the fields accordingly so that it looks like the dialog below:

MO71 Event Dialog

Filling in the MO71 Event Dialog to process an event message

When you are ready right-click the dialog and select ‘start’. This will cause MO71 to start monitoring the queue. Much of what you see in the dialog is self-explanatory but there are one or two things worth mentioning. Firstly the command value you issue can contain inserts. Here we use %o and %m to represent the Object Name and Queue Manager Name respectively. This means that when we get our email, MO71 will tell us which Queue it was that got the event.

You can see that there are a number of other fields in an event which we have left blank. These allow you to do additional processing. For example, you can choose to have any events recorded to the MO71 console or written to a file.

The other thing you can do is re-queue the event message to an alternative Queue. This can be useful if you have multiple applications which would like to process event messages. By daisy chaining in this way each application gets a copy of the message that they can process.

Lastly the status fields, count and last time etc, show how many times this event definition has been ‘fired’. This can be a useful indication that your event definition is picking up the message correctly. In this simple case we would not expect any issues. However, you can make multiple event definitions for the same event types. In this case the Filter and Filter Priority allow you to control the precedence of the event definitions and which one actually processes the message. Looking at the count field is very useful to ensure you have your logic correct.

Anyway, now we have our event definition defined and started all that it takes to cause the email to be sent is to fill a queue. There are many utilities which allow you to do this so I won’t bore you with the details, the one I use is Q (SupportPac MA01) which allows you to very simply fire lots of messages to a queue. So, using this or another tool just verify that you do receive the email you were expecting.

Of course in production you may well want to define a Queue Low event handler as well and perhaps even a Queue Full. You would probably also want to select the Auto Start option. This ensures that the event queue is monitored as soon as MO71 itself is started.

Complex Object attribute monitoring

Event message monitoring is great as long as the event messages that IBM MQ provides match up exactly with your requirements. However, suppose they don’t? Suppose you want more exacting criteria before an alert is raised. Can MO71 help in this too?

Well of course it can otherwise we wouldn’t have mentioned it 🙂

MO71 allows you to repeatedly query a set of Queue Manager object attribute values and process the results. You process the results using what is known as a filter. A filter can be very simple or as complicated as you like. The primary role of a filter is to decide which objects to display to the user in the dialog but you can do lots of other stuff too. One of those things is issue an OS command. So, we now have a mechanism to repeatedly query Queue Manager object attributes and conditionally call an OS command. Just what we need for our last use of blatit.

So, first we need to decide what ‘problem’ we are trying to solve. Let’s say that we want to monitor all queues, except SYSTEM queues and raise an alert if we have a non-zero depth for more than 10 seconds and the open input count is zero. Well, our filter would look something like the following.

@alert := 0;
// Only process non-SYSTEM queues
if (! (queue == "SYSTEM.*"))
{
  // Is this an alert condition?
  if (curdepth & !ipprocs)
  {
    // Set the alert time if we haven't already
    if (!@AlertTime%o) @AlertTime%o := _time;
    // If alert time is over 10 seconds ago, raise alert
    if (@AlertTime%o + 10 < _time) @alert := 1;
  }
  else @AlertTime%o := 0;
}
// Is the alert state active for this object
if (@alert)
{
  // Have we already raised the alert for this object?
  if (!@Alerted%o)
  {
    // Construct and issue the blat command
    @$cmd := 'cmd blatit "Queue ' + queue + ' on ' + _qmname + ' is not getting processed"';
    system(1,@$cmd);
    @Alerted%o := 1;
  }
}
else
{
  // Clear alerted status
  @Alerted%o := 0;
}
@alert;

This may seem complicated at first but the inline comments should make it clear what is happening in the filter. Of course for further information you can refer to the MO71 User Manual. The key point to remember is that the filter is called for each object in the list and returns a Boolean value to determine whether the object is shown in the list. So, all the filter is really doing is remembering the last time it saw an alert situation for each queue. If it hasn’t reported that alert situation then it does so by calling our blatit command file.

Of course typing this in every time would be a pain so this filter should be added to the Filter Manager and given a name, say DepthAlert. Now we can invoke the filter by having a filter of $DepthAlert on any of our queue lists. We can go further in MO71 and define some pre-defined dialogs, with $DepthAlert as the filter and have these dialogs started automatically every time MO71 itself starts. The predefined dialog allows you to specify the refresh frequency. Remember that refreshing the data frequently will make the alerts responsive but will use more resources. As usual with these things it is a compromise.

Summary

So there you have it. Three simple ways where we can have MO71 monitor our MQ definitions or status for us and notify us with an email if something significant happens. Of course this idea is extendable. You don’t need to send an email, you could send an SMS text or whatever your prefer method of notification/logging is. Provided you can write a command file to do the processing you wish then you can drive it from MO71.

Happy monitoring!

Advertisements

The team at MQGem would love to hear what you think. Leave your comments here.

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s