Using ALERTS with MQEV

Our event, accounting and statistics monitoring program, MQEV has a feature to raise alerts. They were initially designed to allow you to notify an MQ administrator of something untoward that has been discovered through an event message or with accounting and statistics data, but they actually serve a number of other purposes. This blog post is going to look at the various different ways you can make use of MQEV alerts.

• Notification to the MQ Administrator
  This is the first, and perhaps most obvious use of alerts in MQEV. For example, upon receiving a Queue Full event, you might raise an alert to notify the MQ administrator of the problem. This could be accompanied by sending an email or text message, for example. In addition, MQEV publishes an MQ message when an alert is raised so that other programs can subscribe to them for speedy notification.

  ****************************************************************
  * Function for processing an event
  ****************************************************************
  func MQEVEvent()
    @objname = event.evobjname
    @objtype = event.evobjtype
    if (event.evreason = PERQDPFULL)
      ************************************************************
      * Alert the MQ Administrator
      ************************************************************
      ADD EVALERT TEXT('Queue High') CATEGORY(VIP) +
          EVOBJNAME('<@objname>') EVOBJTYPE(<@objtype>) +
          SEVERITY(SEVERE) REPLACE
    endif
  endfunc

• Driving a script after some time passes
  This example is something that is also shown in the MQEV manual. In this case you have noticed a situation that, if it remains for too long, is a problem. However, if it clears up quickly, there is no need to trouble the MQ Administrator. The example uses the receipt of a Queue High event. It raises an informational alert with a short expiry time, and if the Queue High status is still present, and has not been cleared with a subsequent Queue Low event, the expiration of the alert will drive code to notify the MQ administrator of the problem.

  ****************************************************************
  * Function for processing an event
  ****************************************************************
  func MQEVEvent()
    @objname = event.evobjname
    @objtype = event.evobjtype
    if (event.evreason = PERQDPHI)
      ************************************************************
      * Add a temporary alert that will expire in 60 seconds
      ************************************************************
      ADD EVALERT TEXT('Queue High') CATEGORY(QHIGHTEMP) +
          EVOBJNAME('<@objname>') EVOBJTYPE(<@objtype>) +
          SEVERITY(INFO) RETINTVL(60) REPLACE
    endif
    if (event.evreason = PERQDPLO)
      ************************************************************
      * Remove both alert types, temp & final, wildcarded CATEGORY
      ************************************************************
      REMOVE EVALERT TEXT('Queue High') CATEGORY(QHIGH*) +
             EVOBJNAME('<@objname>') EVOBJTYPE(<@objtype>)
    endif
  endfunc
  ****************************************************************
  * Function called when an alert expires
  ****************************************************************
  func MQEVAlertExpire()
    if (alert.category = 'QHIGHTEMP')
      @objname = alert.evobjname
      @objtype = alert.evobjtype
      ADD EVALERT TEXT('Queue High') CATEGORY(QHIGH) +
          EVOBJNAME('<@objname>') EVOBJTYPE(<@objtype>) +
          SEVERITY(WARN) REPLACE
    endif
  endfunc

• Drive Code repeatedly
  An extension of the above idea, is to use expiring alerts to drive code on a regular basis to check on something, perhaps a behaviour in MQ that isn’t able to be monitored by event messages. After checking the attribute, a new expiring alert, based on the one that just expired, is created, and round it goes again. The MonitorFunction() might also create alerts of the first type.

  ****************************************************************
  * Function: addEVAlert
  ****************************************************************
  func addEVAlert(Text, Category, RetIntvl)
    =mqev
    ADD EVALERT TEXT('<@Text>') CATEGORY('<@Category>') +
        RETINTVL(<@RetIntvl>) EVQMGR(<_qmgr>) +
        SEVERITY(INFO) REPLACE
    =mqsc
  endfunc
  ****************************************************************
  * Function called when we connect
  ****************************************************************
  func MQEVConnected()
    **************************************************************
    * Initialise the expiring alerts that drive the monitoring fns
    **************************************************************
    addEVAlert("Delay: 2 Hours",   "DELAY.2HRS",  2 * 60 * 60)
  endfunc
  ****************************************************************
  * Function called when we an alert expires
  ****************************************************************
  func MQEVAlertExpire()
    if (alert.CATEGORY = 'DELAY.2HRS')
      MonitorSomething(...)
      * Now remake the same alert *
      addEVAlert(alert.TEXT, alert.CATEGORY, alert.RETINTVL)
    endif
  endfunc

• Drive some code in an ad hoc manner
  Again, in an extension to the above model, you could have code to run that is driven by an expiring alert, and could be driven in an ad hoc manner by an administrative command creating an alert with a very short expiry.

  ****************************************************************
  * Function called when we an alert expires
  ****************************************************************
  func MQEVAlertExpire()
    if (alert.CATEGORY = 'DRIVE.NOW')
      CheckSomething(...)
    endif
  endfunc

  This code could be driven by an MQ administrator using the following command:

  ADD EVALERT TEXT('Drive code') CATEGORY(DRIVE.NOW) +
      RETINTVL(1) SEVERITY(INFO) REPLACE

• State Variable
  When monitoring for situations using event messages or even issuing direct MQ commands, you can raise an alert for an MQ Administrator to take notice of what has been discovered. If the same situation is seen again, and the MQ Administrator has not yet noticed the previous alert, you may decide that you don’t need to contact him again. The existence of the alert in the system marks that he has already been notified. Once the MQ Administrator deals with the alert, he can delete it. The lack of an alert in the system would then mean that the script would notify him again if the situation subsequently arose.

  ****************************************************************
  * Function: AlertUser 
  * Purpose: This function will alert the user in whichever way is
  *          needed, for example, raise an alert, or send an email
  ****************************************************************
  func AlertUser(objname, objtype, category, alert_text)
    =mqev
    DIS EVALERT CATEGORY('<@category>') +
        EVOBJTYPE(<@objtype>) EVOBJNAME('<@objname>') +
        EVQMGR(<_qmgr>)
    if (_matches = 0)
      ************************************************************
      * Send email to notify MQ Administrator *
      ************************************************************
      system('email.bat "' + @alert_text + '"')
    else
      @id = EVALERT
      REMOVE EVALERT(<@id>)
    endif
    **************************************************************
    * Always add a new replacing alert so that the date, and any
    * new text in the alert, is updated
    **************************************************************
    ADD EVALERT TEXT('<@alert_text>') CATEGORY('<@category>') +
        EVOBJTYPE(<@objtype>) EVOBJNAME('<@objname>') +
        EVQMGR(<_qmgr>)
    =mqsc
  endfunc

As you can see, MQEV and its alerts are very flexible. Alerts can notify an MQ Administrator of an event within your MQ estate, and in addition can be used to control when those alerts might be raised.

---

The MQEV product is free for you to download and try out until June 30th 2020. You don’t even need to register any details to play with it. You can download it from our website.

All we ask in return is that in a week or two when you’ve played with it you send us an email telling us what you liked and what you didn’t like about it and perhaps what features you’d like to see in the future.

We look forward to hearing your comments. Please send them to support@mqgem.com.

MQEV Topologies

MQSCX and MO71 provide admin for MQEV

MQEV uses a couple of queues in order to operate. A command queue (CQ) to receive inquiries from tools such as MO71 and MQSCX; and a persistence queue (PQ) where compressed event, accounting and statistics data is held once received from the IBM MQ event queues (EQ).

The program itself can run using a client or local bindings connection. This provides a number of choices for the topology arrangement that you can use with MQEV. In this post we discuss our two main recommended topologies.

All queues local to MQEV

This is the ideal approach. It uses a local connection for access to all the queues involved.

In this topology, MQEV runs as a locally bound application on the same machine as the queue manager being monitored. The event queues (EQ) and the persistence queue (PQ) are hosted on the same queue manager. Commands are sent to MQEV via the command queue (CQ), by administrative tools such as MQSCX and MO71. These tools may be connected either locally or over a client connection, just as they do for the IBM MQ Command Server.

In the diagram below, the MO71 administrative tool is shown in the centre of the diagram on a separate machine to the queue managers, but of course it is possible that the machine where MO71 is running may also host one of the monitored queue managers.

To run the MQEV program in this way, issue a command of the following format (where MQG1 is the monitored queue manager):

mqev -m MQG1

MQEV Topology: MQEV runs local to each queue manager to be monitored

Separate state QMgr for the persistence queues

There may be times when running MQEV local to the monitored queue manager is not appropriate, or even possible. For example, MQEV is not currently available as a z/OS program, and it clearly can’t run locally on the MQ Appliance. Additionally, installations may have rules about what programs can run local to a queue manager.

In such cases, MQEV can connect as a client to the monitored queue manager, but maintain its persistent state on a local queue manager, ideally one dedicated to run your MQEV instances. This state queue manager holds the persistence queues (PQ), one for each monitored queue manager, and the event queues (EQ) are read by MQEV over a client connection. Commands are sent to MQEV via the command queue hosted on each monitored queue manager as before, also read by MQEV over the client connection. Administrative tools such as MQSCX and MO71 connect to the queue managers just as they do for the IBM MQ command server. They do not connect to the MQEV state queue manager.

To run the MQEV program in this way, issue a command of the following format (where STATEQM is the state queue manager, and -l indicates the connection to MQG1 is over a client):

mqev -m MQG1 -l -s STATEQM

The client connection in this case should be configured with the use of a CCDT file, by defining a CLNTCONN channel definition with a QMNAME of MQG1.

MQEV Topology: Dedicated state queue manager for MQEV persistence queues

If you have other topology requirements that are not covered by the above two setups, we would be interested to hear from you.

---

The MQEV product is free for you to download and try out until June 30th 2020. You don’t even need to register any details to play with it. You can download it from our website.

All we ask in return is that in a week or two when you’ve played with it you send us an email telling us what you liked and what you didn’t like about it and perhaps what features you’d like to see in the future.

We look forward to hearing your comments. Please send them to support@mqgem.com.

First Steps with MQEV and MQ Accounting and Statistics

Our newest product MQEV, can be used to collect, process, store and search the data generated by your IBM MQ Queue Manager as Accounting and Statistics messages. In this post, we will take you through the first steps of using MQEV to look at Accounting and Statistics messages.

We assume that you have downloaded the MQEV program from our website, and also grabbed the latest version of either MQSCX or MO71 to interact with MQEV. You don’t need to have a licence file for either of these products in order to use them with MQEV – they are covered by the MQEV licence, and will work for free without any licence until June 30th 2020. Installation of MQGem products is really easy – just unzip/untar into a location that is in your path and off you go.

Make sure your queue manager is running and then run the provided script file config.mqx to define the two queues that the MQEV product requires in order to work (MQGEM.MQEV.COMMAND.QUEUE and MQGEM.MQEV.DATA.QUEUE). You may have already done this step if you have used MQEV with MQ Event messages.

Now run the MQEV program with the following command (assuming your queue manager is called MQG1).

mqev -m MQG1

Now MQEV is running, but it’s not capturing any accounting and statistics yet – first you have to tell it what you’re interested in. First turn them on in your queue manager with the following MQSC commands (or their equivalent action in your favourite GUI). I’m just enabling MQI statistics, and Queue accounting for my queue named Q1. I’ve shortened the Statistics Interval a bit too so you don’t have to wait as long to see some data. You can turn on Queue accounting (and queue statistics) for all queues at once using the queue manager attribute of the same name, but for now, I’m just looking at a single queue. There are in fact five different types of accounting and statistics for you to use. Read more about them here.

ALTER QMGR STATMQI(ON) STATINT(900)

ALTER QLOCAL(Q1) ACCTQ(ON)

Now your queue manager will generate some accounting and statistics, and the next step is to tell MQEV to listen out for them. You can do this using a command line tool such as MQSCX (in =mqev mode) or using the MO71 GUI. Open one of the twisties below to see instructions.

Command Line: MQSCX

Issue the following commands:

=mqev
RESUME EVQ(SYSTEM.ADMIN.ACCOUNTING.QUEUE)
RESUME EVQ(SYSTEM.ADMIN.STATISTICS.QUEUE)

If you now issue the following command your will see that the two queues you resumed above, now show STATUS(ACTIVE).

DISPLAY EVQ(*)

Using MO71 GUI

Right click on the queue manager in the main window and from the context menu choose MQ EV Events -> MQ Event Queue List …
Press the “Refresh” button to populate the list.
You will be presented with a list of the known event queue names (these are pre-configured to MQEV for you, but all in a suspended state so that you control when to start reading from them). Select the following two queues, the right click and choose “Resume” from the context menu.

• SYSTEM.ADMIN.ACCOUNTING.QUEUE
• SYSTEM.ADMIN.STATISTICS.QUEUE

Resume context menu on a list of MQEV event queues

You will see the dialog list automatically refresh and show that your two queues now have a status of “Active” instead of “Suspended”.

Now you are going to generate some accounting data by connecting an application that makes use of your queue. You can use any MQ application you like, but for simplicity, I’m just using the IBM-supplied sample amqsput.

amqsput Q1 MQG1

Accounting information is written out by the IBM MQ Queue Manager when an application disconnects. So once you have finished putting some messages using an MQ application, make sure to end the application so that it disconnects, and then you will be able to see the data. Take a look at accounting data now. Open one of the twisties below to see instructions.

Command Line: MQSCX

Issue the following commands:

=mqev
DISPLAY ACCTQ(*)

You will see output something like this:-

[09:30:45] =mqev

Commands directed to MQEV

[09:30:53] DISPLAY ACCTQ(*)

ACCTQ(Q1) INTVLEND(2020-03-09 09:30:44 (Local)) ALLPUT(8)

Total display responses – Received:1

MQEV:MQG1>

Using MO71 GUI

Right click on the queue manager in the main window and from the context menu choose MQEV Acct & Stat -> MQ Accounting Queue List …
Press the “Refresh” button to populate the list.
You’ll see one line in the list like the screenshot below.

MO71 showing accounting data captured by MQEV

This just shows a summary of the accounting data. Now let’s look at all the details in this record.

Command Line: MQSCX

Repeat your last command and add ‘ALL’ to the end to see the full data in the record.

DISPLAY ACCTQ(*) ALL

You’ll see output like the following.

[09:31:51] DISPLAY ACCTQ(*) ALL

ACCTQ(Q1) QTYPE(QLOCAL) DEFTYPE(PREDEFINED)

APPLNAME(E:\mqm9130\bin64\amqsput.exe) USERID(mqgemusr)

CONNID(414D51434D51473120202020202020207E20475E01663522) PID(87308)

TID(1) EVSTREAM($ACCTQ) EVQMGR(MQG1)

INTVLSTA(2020-03-09 09:30:43 (Local)) INTVLEND(2020-03-09 09:30:44 (Local))

OPENTI(2020-03-08 20:30:40 (UTC)) CLOSETI(2020-03-08 20:30:44 (UTC))

RECORDS(1) OPENCNT(1) CLOSECNT(1) ALLPUT(8)

PUT(8) PUTNP(8) PUTBYTE(24) PUTBYTENP(24)

PUTMINBYTENP(3) PUTMAXBYTE(3) PUTMAXBYTENP(3)

Total display responses – Received:1

MQEV:MQG1>

In this output you can see that the application opened the queue once (OPENCNT), and put 8 non-persistent messages(PUTNP), with a total of 24 bytes of data (PUTBYTE). You’re also shown identifying fields about the application, its name, the user id that ran it, the Connection ID and process and thread IDs.

Using MO71 GUI

Double click on the record in the list dialog to take a look at the details.

MO71 display of an individual accounting record

On the ‘General’ tab you’ll see information identifying the application, its name, the user id that ran it, the process and thread ID and the Connection ID. You’l also see that it opened the queue once.

On the ‘Messaging’ tab you’ll see the number of puts it did, in my example there were 8 non-persistent puts.

On the ‘Bytes’ tab you’ll see how many bytes of message data it put in total.

By now, your Statistics Interval should have passed, and there will be some Statistics data to look at too. Statistics data is recorded by the IBM MQ Queue Manager on a regular interval, as defined by the STATINT queue manager attribute. These statistics are for all the MQI calls made across the whole queue manager, not just your one application using queue Q1. Open one of the twisties below to see instructions.

Command Line: MQSCX

Issue the following command:

=mqev
DISPLAY STATMQI(*)

You will see output something like this:-

[10:22:22] DISPLAY STATMQI(*)

STATMQI($STATMQI) INTVLEND(2020-03-09 10:22:17 (Local)) ALLPUT(392)

GET(279)

Total display responses – Received:1

MQEV:MQG1>

Repeat the command with ‘ALL’on the end to see even more details.

Using MO71 GUI

Right click on the queue manager in the main window and from the context menu choose MQEV Acct & Stat -> MQ Statistics MQI List …
Press the “Refresh” button to populate the list.
You’ll see one line in the list like the screenshot below.

MO71 showing statistics data captured by MQEV

Double click on the line, to see the full details of the record.

There are five different types of Accounting and Statistics data written by an IBM MQ Queue Manager, and there is an MQEV command for displaying each type.

Command Line: MQSCX	Using MO71 GUI
DISPLAY ACCTMQI(*) DISPLAY ACCTQ(*) DISPLAY STATCHL(*) DISPLAY STATMQI(*) DISPLAY STATQ(*)

By enabling Accounting and Statistics on your queue manager, MQEV can collect and store the data for your queues, and you can display the activity at any time, if necessary going back several days or weeks. You can of course, even graph the activity to more easily show how it changes over time. Watch this space for more on graphing.

Try out enabling some of the other types of IBM MQ Accounting and Statistics, collecting them with MQEV (both queues that you need to read from have been resumed in MQEV by following this post), and displaying the data you receive.

Look out for more posts about how you can manipulate the different views of the data that Accounting and Statistics gives you.

---

The MQEV product is free for you to download and try out until June 30th 2020. You don’t even need to register any details to play with it. You can download it from our website.

All we ask in return is that in a week or two when you’ve played with it you send us an email telling us what you liked and what you didn’t like about it and perhaps what features you’d like to see in the future.

We look forward to hearing your comments. Please send them to support@mqgem.com.

Act on MQ Event messages

Some event messages tell you something that you might like to act upon immediately. Others you might just want to keep an eye on, for example channel problems where channel retry will probably cover it.

In this blog post, I’ll demonstrate an example of how you might use MQEV to immediately act upon an event message. In our demonstration today, we have a cluster of queue managers with a number of hosted queues called CLUS.Q1. Using a similar tactic to that used by the The Cluster Queue Monitoring sample program (AMQSCLM), we’re going to lower the CLWLPRTY value of a cluster queue when it becomes a less useful choice, in this case when it hits the Queue-High state.

Turning on Queue Depth Events in IBM MQ

On each cluster queue manager, we would turn on Queue Depth events with the following commands, and set an initial value of CLWLPRTY to allow it to be lowered.

ALTER QMGR PERFMEV(ENABLED)

ALTER QLOCAL(CLUS.Q1) MAXDEPTH(10000) +
      QDEPTHHI(40) QDEPTHLO(10) +
      QDPHIEV(ENABLED) +
      CLWLPRTY(5)

Explanation of Queue Depth settings

In this example, rather than following the usual Queue Depth High of 80% and Queue Depth Low of 20%, with, say a maximum depth of 5000, I’ve doubled the maximum depth to 10000, and set the percentages so I still get a Queue High at 4000, and Queue Low at 1000. This gives me much more head-room before I hit Queue Full, but with 4000 messages on the queue, that’s still a hefty backlog to work through.

Turning on Queue Depth Events in MQEV

To ensure MQEV is processing performance events, issue the following command to MQEV.

RESUME EVQ(SYSTEM.ADMIN.PERFM.EVENT)

Processing Queue Depth Events in MQEV

Now we’re going to add some instructions to MQEV, to tell it what do to when it receives Queue Depth events. This is done by using the MQEVEvent() function in the mqev.mqx script. In the script below it tests for the event type, looking for performance events (event.evtype = PERFM), and then displays the queue that the event was all about, since we need to tell whether it was a cluster queue, and if so what it currently has as a value for CLWLPRTY.

*********************************************************************
* Function for processing an event                                  *
*********************************************************************
func MQEVEvent()
  if (event.evtype = PERFM)
    =mqsc
    @qname = event.evoobjname
    DISPLAY QLOCAL(<@qname>) ALL
    if (CLUSTER OR CLUSNL)
      ...
    endif
  endif
  endfunc

If it finds that this is indeed a cluster queue, then it will do some adjusting of the CLWLPRTY value based on whether this is Queue High, Queue Low, or Queue Full.

If you have a good naming convention for your queues, you could avoid the step of displaying the queue, and rely on your naming convention pattern to inform you that it is a cluster queue.

*********************************************************************
* Function for processing an event                                  *
*********************************************************************
func MQEVEvent()
  if (event.evtype = PERFM)
    =mqsc
    @qname = event.evoobjname
    DISPLAY QLOCAL(<@qname>) ALL
    if (CLUSTER OR CLUSNL)
      if (event.evreason = PERQDPHI)
        ALTER QLOCAL(<@qname>) CLWLPRTY(4)
      endif
      if (event.evreason = PERQDPFULL)
        ALTER QLOCAL(<@qname>) PUT(DISABLED)
      endif
      if (event.evreason = PERQDPLO)
        ALTER QLOCAL(<@qname>) CLWLPRTY(5) PUT(ENABLED)
      endif
    endif
  endif
endfunc

When it hits a Queue High state, it lowers the CLWLPRTY by one, making it a less good choice than instances of the queue on other queue managers. Of course, if all instances of the queue are lowered in the same way, they all become equally appropriate choices again. You might, in addition, want to raise an alert to ensure the MQ System administrator is aware of the situation.

When it hits a Queue Full state, it changes the queue to be PUT(DISABLED). This is because that attribute is one of the ones flowed round the cluster, and it will stop this queue from being chosen as a result.

When it hits a Queue Low state, it sets the CLWLPRTY back to 5 and re-enables the queue for putting again.

These are just example actions to show you how you could act upon an event message at the moment it arrives on the event queue. You can of course, issue any command you like. You could for example, start more instances of your application, perhaps by running a script using the MQSCX system() function, or by defining it as an MQ SERVICE object and starting that.

---

The MQEV product is free for you to download and try out until June 30th 2020. You don’t even need to register any details to play with it. You can download it from our website.

All we ask in return is that in a week or two when you’ve played with it you send us an email telling us what you liked and what you didn’t like about it and perhaps what features you’d like to see in the future.

We look forward to hearing your comments. Please send them to support@mqgem.com.

Does MO71 log the commands it issues?

This is a question that we get asked quite regularly, so we thought it worth writing up the answer as a blog post.

If you’re asking this question, there are two reasons that immediately come to mind that might be behind your question:-

• You just want some sort of documentary log, a history of what you have done to look back on.
• You need a log of commands issued for audit reasons.

However, applications logging their own activity have a number of disadvantages. For example, if the log file were on your own machine, then you have to manage those files, house-keep them as they get old enough to delete, etc. Flat log files are not very easy to search through, say to find out what happened last weekend, or all the activity associated with a particular object.

If you want something for audit reasons, using a log file on the user machine has a number of additional problems:-

• Firstly, it is not audit proof because
  • It is easy to circumvent
  • It is incomplete since other tools might be used to issue commands to the queue manager
  • The user could switch off the logging feature in their tool, or install a new version of it to avoid the logging configuration
• Secondly, it is hard to collect the logs from all the different users’ machines to have the complete picture.
• Thirdly, if individual applications are creating the log file, e.g MO71, runmqsc, MQ Explorer and so on, then you get a myriad of different log file formats.

To solve all of the above, IBM MQ Queue Managers have a feature called Command Events. When enabled these result in an event message being emitted by the queue manager for any command issued (with a tweak to avoid DISPLAY commands if you wish – see EVENT Switches). This is a far better solution than logging in the user-facing tool (e.g. MO71), for reasons such as:-

• Central collection is better because it is easier to obtain a complete record, since the records are not segregated across many user machines
• It cannot be subverted by non-privileged users
• The format is consistent regardless of the tool used by the user
• No updates are required to any tool, the logging capability is available immediately for all applications

If all you want is something to pass an audit, command and configuration events might be enough. But if you want a record for diagnostic purposes there are other things you might wish to add to the record of what happened, such as:-

• Channel problems via Channel events
• Queues High or Full via Performance events
• Queue Service Interval High via Performance events
• Queue Manager Fail-overs via Start/Stop events

We have been recommending using Command (and Configuration) events rather than logging at the user-side MQ administration tool, for many years. Previously the problem with this recommendation was the collection and searching of the event data. However, we have recently released a new product, MQEV, which will help you in this regard. MQEV will collect, process and store your event data, and make searching and report building very simple. You can now easily get the information from the event messages that your IBM MQ queue manager is designed to emit. If required you can process the events real-time, and make changes immediately, or even raise alerts to your MQ administrators for follow-up.

---

Lumberjack image courtesy of vectorolie at FreeDigitalPhotos.net

Configuration Change Events in MQEV

When a change is made to your queue manager’s configuration, you can be delivered event messages notifying you of the change. These event messages are called Configuration Events.

When the notification being made is about a change in the configuration (rather than the creation or deletion of an object), then you are delivered two event messages; a before image and an after image of the object being changed. In order to determine what actually changed, the two event messages must be compared to find the differences. Both events, the before and the after, contain all the attributes of the object. To find what changed, all attributes in each event must be compared. In the screenshot below, two browse windows showing the two event messages are shown side by side.

Manually Comparing two Config Events

Using MQEV to display your event messages instead of simply browsing them as formatted PCF messages can improve your understanding of what happened hugely.

The default display output from the MQEV command server will include a summary of the change event.

SUMMARY(Config - Change Object - Queue:Q1 - MAXDEPTH[5000 -> 10000])

MQEV has compared the two event messages that were emitted by the queue manager, and discovered what the change was and provided a very helpful summary.

You can, of course, look at the whole event message if you need to. For example in MQSCX, you can use the following command to show all the attributes of the object. MQEV saves on storage space by only storing one copy of the attributes that haven’t changed. Essentially, the two configuration events are combined inside MQEV into one item.

DISPLAY EVENTS(*) EVTYPE(CONFIG) EVOBJNAME(Q1) ALL

MQSCX Extended MQSC Program – Version 9.1.0

Licenced to Paul Clarke

Licence Location: Home Office

[16:44:32] DISPLAY EVENTS(*) EVTYPE(CONFIG) EVOBJNAME(Q1) ALL

EVQMGR(MQG1) EVENTS($EVENTS) EVTIME(2020-02-07 11:55:35 (Local))

EVREASON(CFGCHGOBJ) EVTYPE(CONFIG) EVUSERID(mqgemusr) EVOBJNAME(Q1)

EVOBJTYPE(QUEUE) EVENTID(00000286) CFHCMD(43) CFHREASON(2368)

SUMMARY(Config – Change Object – Queue:Q1 – MAXDEPTH[5000 -> 10000])

BEFORE:

EVENTUSER(mqgemusr)

EVSID(1D010105000000000005150000001AFA5FFE70975006C3A1C633E903000000000000000000000000)

EVORIGIN(MSG)

EVACCTTK(160105150000001AFA5FFE70975006C3A1C633E903000000000000000000000B)

EVAPPLID( ) EVAPPLTYPE(WINDOWSNT)

EVAPPLNAME(d:\nttools\mqscx.exe) EVAPPLORIG( ) OBJTYPE(QUEUE)

QUEUE(Q1) DESCR( ) PROCESS( ) BOQNAME( )

INITQ( ) TRIGDATA( ) CLUSCHL( ) CUSTOM( )

CLUSTER( ) CLUSNL( ) CRDATE(2019-06-19) CRTIME(06.29.19)

ALTDATE(2019-12-27) ALTTIME(09.24.54) GET(ENABLED) PUT(ENABLED)

DEFPRTY(0) DEFPSIST(NO) MAXDEPTH(5000) MAXMSGL(4194304)

BOTHRESH(0) SHARE(1) DEFSOPT(SHARED) HARDENBO(1)

MSGDLVSQ(PRIORITY) RETINTVL(999999999) USAGE(NORMAL) TRIGCTL(NOTRIGGER)

TRIGTYPE(FIRST) TRIGDEPTH(1) TRIGMPRI(0) QDEPTHHI(80)

QDEPTHLO(20) QDPMAXEV(ENABLED) QDPHIEV(DISABLED) QDPLOEV(DISABLED)

QSVCINT(999999999) QSVCIEV(DISABLED) DISTL(NO) NPMCLASS(NORMAL)

STATQ(ON) ACCTQ(ON) MONQ(HIGH) SCOPE(QMGR)

DEFBIND(OPEN) CLWLRANK(0) CLWLPRTY(0) CLWLUSEQ(QMGR)

DEFPRESP(SYNC) DEFREADA(NO) PROPCTL(COMPAT) IMGRCOVQ(QMGR)

DEFTYPE(PREDEFINED) QTYPE(QLOCAL)

AFTER:

ALTDATE(2020-02-06) ALTTIME(22.55.35) MAXDEPTH(10000)

Total display responses – Received:1

MQEV:MQG1>

If you don’t want to see all the attributes, you can add DISTYPE(CONDENSE) to the end of your command in MQSCX, to get output like the following, where the BEFORE: section just shows the attributes that are changed.

[16:46:37] DISPLAY EVENTS(*) EVTYPE(CONFIG) EVOBJNAME(Q1) ALL DISTYPE(CONDENSE)

EVQMGR(MQG1) EVENTS($EVENTS) EVTIME(2020-02-07 11:55:35 (Local))

EVREASON(CFGCHGOBJ) EVTYPE(CONFIG) EVUSERID(mqgemusr) EVOBJNAME(Q1)

EVOBJTYPE(QUEUE) EVENTID(00000286) CFHCMD(43) CFHREASON(2368)

SUMMARY(Config – Change Object – Queue:Q1 – MAXDEPTH[5000 -> 10000])

BEFORE:

MAXDEPTH(5000)

AFTER:

MAXDEPTH(10000)

Total display responses – Received:1

MQEV:MQG1>

If you’re using MO71 to view events, you’ll notice that MO71 has a new type of dialog designed especially for these change events. Where you might well be used to seeing the dialog show the labels on the left hand side and the values to the right; now the right hand side has two columns. The left shows all the attributes, and the right shows the changes, with a background highlight colour (in case the change made was blanking out a field). You can control this highlight colour from your View->Set Colours.. menu in the usual manner.

MO71 dialog showing a change event

So, now you know what we mean when we say MQEV consolidates, and summarises change events. MQEV doesn’t just capture them and let you look at them later, it helps you to understand more easily what the events are telling you.

---

MQEV is free for you to download and try out until June 30th 2020. You don’t even need to register any details to play with it. You can download it from our website.

All we ask in return is that in a week or two when you’ve played with it you send us an email telling us what you liked and what you didn’t like about it and perhaps what features you’d like to see in the future.

We look forward to hearing your comments. Please send them to support@mqgem.com.

Retaining Events with MQEV

How long should you keep event messages for?

That’s one of those “It depends” questions.

• Is it worth keeping command events for DISPLAY commands at all?
• Is it worth keeping channel retrying events once the channel is up and running again?
• Would you keep “not authorized” events for longer than logger events?

MQEV stores your IBM MQ event data on streams. By default all events are stored on the $EVENTS stream which has a retention interval of 90 days. Streams are your control point for how long to hold onto this data. You might decide that authority events should be kept for much longer, but that channel events can be kept for a shorter time. In which case you might create some new streams thus:-

DEFINE EVSTREAM(CHANNEL) TYPE(EVENTS) RETINTVL(30)
DEFINE EVSTREAM(SECURITY) TYPE(EVENTS) RETINTVL(120)

And then send different event types to their appropriate streams with the following in your MQEVEvent() function in the mqev.mqx script.

*********************************************************************
* Function for processing an event                                  *
*********************************************************************
func MQEVEvent()
  if (event.evtype = AUTHOR)
    _stream = "SECURITY"
  endif;
  if (event.evtype = CHANNEL)
    _stream = "CHANNEL"
  endif;
endfunc

You can still display all your events across these different streams with the usual DISPLAY EVENTS(*) command or the MO71 events list dialog, but if you need to, you can now also filter them by the stream that they are stored in.

MQEV will take care of deleting these events once they pass the retention interval defined on the stream they are stored in.

Of course, this is a very simplistic example. You could separate events out further, for example, by a specific reason. Some channel events might be worth keeping with the authority events, for example “Channel is Blocked” events. Let’s make a small change to the MQEVEvent() function to achieve that.

*********************************************************************
* Function for processing an event                                  *
*********************************************************************
func MQEVEvent()
  if (event.evtype = AUTHOR)
    _stream = "SECURITY"
  endif;
  if (event.evtype = CHANNEL)
    if (event.cfhreason = const.MQRC_CHANNEL_BLOCKED)
      _stream = "SECURITY"
    else
      _stream = "CHANNEL"
    endif;
  endif;
endfunc

And you are not limited there – you can use any data in the event message to determine which stream it should be stored on. For example, you might have a very important application whose retention interval is much higher than everything else, and so make use of the object name in the event, in order to save it off to the appropriate stream. This is where having a good naming convention helps greatly.

First Steps with MQEV and MQ Events

Our newly released product MQEV, can be used to collect, process, store and search the data generated by your IBM MQ Queue Manager as MQ Event messages. In this post, we will take you through the first steps of using MQEV to look at event messages.

We assume that you have downloaded the MQEV program from our website, and also grabbed the latest version of either MQSCX or MO71 to interact with MQEV. You don’t need to have a licence file for either of these products in order to use them with MQEV – they are covered by the MQEV licence, and will work for free without any licence until June 30th 2020. Installation of MQGem products is really easy – just unzip/untar into a location that is in your path and off you go.

Make sure your queue manager is running and then run the provided script file config.mqx to define the two queues that the MQEV product requires in order to work (MQGEM.MQEV.COMMAND.QUEUE and MQGEM.MQEV.DATA.QUEUE).

Now run the MQEV program with the following command (assuming your queue manager is called MQG1).

mqev -m MQG1

Now MQEV is running, but it’s not capturing any events yet – first you have to tell it what you’re interested in. Let’s assume you want to process command events and configuration events to begin with. (I’ve chosen these because they are the easiest events to generate so you can immediately see some data in MQEV). First turn them on in your queue manager with the following MQSC command (or it’s equivalent action in your favourite GUI).

ALTER QMGR CMDEV(NODISPLAY) CONFIGEV(ENABLED)

(Wondering about CMDEV(NODISPLAY)? Read more here).

Now your queue manager will generate command and configuration events, and the next step is to tell MQEV to listen out for them. You can do this using a command line tool such as MQSCX (in =mqev mode) or using the MO71 GUI. Open one of the twisties below to see instructions.

Command Line: MQSCX

Issue the following commands:

=mqev
RESUME EVQ(SYSTEM.ADMIN.COMMAND.EVENT)
RESUME EVQ(SYSTEM.ADMIN.CONFIG.EVENT)

If you now issue the following command your will see that the two queues you resumed above, now show STATUS(ACTIVE).

DISPLAY EVQ(*)

Using MO71 GUI

Right click on the queue manager in the main window and from the context menu choose MQ EV Events -> MQ Event Queue List …
Press the “Refresh” button to populate the list.
You will be presented with a list of the known event queue names (these are pre-configured to MQEV for you, but all in a suspended state so that you control when to start reading from them). Select the following two queues, the right click and choose “Resume” from the context menu.

• SYSTEM.ADMIN.COMMAND.EVENT
• SYSTEM.ADMIN.CONFIG.EVENT

Resume context menu on a list of MQEV event queues

You will see the dialog list automatically refresh and show that your two queues now have a status of “Active” instead of “Suspended”.

The first thing you will notice is that you are immediately given some events. This is because you just issued a command to MQ and made a change to the configuration of your queue manager – just by turning on the event switches! Take a look at these events now. Open one of the twisties below to see instructions.

Command Line: MQSCX

Issue the following commands:

=mqev
DISPLAY EVENTS(*)

You will see output something like this:-

[20:36:55] =mqev

Commands directed to MQEV

[20:37:07] DISPLAY EVENTS(*) EVENTID

_________________________________________________________________________________

EVQMGR(MQG1) EVTIME(2020-01-03 20:36:10 (Local)) EVENTID(00000002)

SUMMARY(Config – Change Object – Qmgr:MQG1 – CONFIGEV[DISABLED -> ENABLED] CMDEV[

DISABLED -> NODISPLAY])

_________________________________________________________________________________

EVQMGR(MQG1) EVTIME(2020-01-03 20:36:10 (Local)) EVENTID(00000001)

SUMMARY(Command – Change Qmgr – Qmgr:MQG1))

_________________________________________________________________________________

Total display responses – Received:2

MQEV:MQG1>

Using MO71 GUI

Right click on the queue manager in the main window and from the context menu choose MQ EV Events -> MQ Event List …
Press the “Refresh” button to populate the list.
You’ll see two events in the list like the screenshot below.

MO71 showing events captured by MQEV

This just shows a summary of the event details. Now let’s look at all the contents of one of the events.

Command Line: MQSCX

Issue the following commands to look at the Change Qmgr configuration event from the list.

DISPLAY EVENTS($EVENTS) EVQMGR(MQG1) EVENTID(2) DISTYPE(CONDENSE) ALL

You’ll see output like the following. This display type is a good way to look at change events because you don’t get swamped with all the rest of the attributes from the object (a queue manager object has quite a lot of attributes), and instead are only shown the ones that have changed. Experiment without the display type to see the difference.

[20:47:07] DISPLAY EVENTS($EVENTS) EVQMGR(MQG1) EVENTID(2) DISTYPE(CONDENSE) ALL

EVQMGR(MQG1) EVENTS($EVENTS) EVTIME(2020-01-03 20:36:10 (Local))

EVREASON(CFGCHGOBJ) EVTYPE(CONFIG) EVUSERID(mqgemusr) EVOBJNAME(MQG1)

EVOBJTYPE(QMGR) EVENTID(00000002) CFHCMD(43) CFHREASON(2368)

SUMMARY(Config – Change Object – Qmgr:MQG1 – CONFIGEV[DISABLED -> ENABLED] CMDEV[

DISABLED -> NODISPLAY])

BEFORE:

CONFIGEV(DISABLED) CMDEV(DISABLED)

AFTER:

CONFIGEV(ENABLED) CMDEV(NODISPLAY)

Total display responses – Received:1

MQEV:MQG1>

In this output you can see two small sections at the end, a “BEFORE:” section and an “AFTER:” section showing the values of the attributes that have been changed.

Using MO71 GUI

Double click on the “Config – Change Object” event to take a look at the details. This dialog will look very similar to the Queue Manager object dialog that you might well be familiar with from making changes to your queue manager. It has the same recognisable set of tabs along the top. Switch to the “Monitoring” tab and you’ll see the attributes that were changed and are reported in this event.

MO71 display of a single configuration change event from MQEV

Now create some other objects, or make changes to pre-existing objects to see more examples of command and configuration events and use the above commands or variations on them to look at the details.

We chose command and configuration events because they are so easy to generate. Other events are sometimes a bit harder to cause to happen, so to make it easier for you to look at what these other events contain, we have also provided a QLOAD / dmpmqmsg file containing a number of other events that you can just load onto one of your event queues (for example one of the two that MQEV is already reading) and then have a good old look around them.

Issue the following command:

qload -m MQG1 -o SYSTEM.ADMIN.COMMAND.EVENT -f TestEventMsgs.qld -CI

If you’re not a QLOAD licensee, you can, in this case, use dmpmqmsg with the same parameters.

---

The product is free for you to download and try out until June 30th 2020. You don’t even need to register any details to play with it. You can download it from our website.

All we ask in return is that in a week or two when you’ve played with it you send us an email telling us what you liked and what you didn’t like about it and perhaps what features you’d like to see in the future.

We look forward to hearing your comments. Please send them to support@mqgem.com.

Announcing the MQGem Event Processor!

MQGem Software is pleased to announce that it has just released a new product, MQEV, which makes processing Event, Accounting and Statistics messages a breeze. What’s more the product is absolutely free to use for the first 6 months to enable you to verify that it is useful to you.

Many installations don’t make full use of the Event, Accounting and Statistics messages that MQ can generate because the messages they generate are hard to analyse. This is a huge waste of valuable information which could be very useful in your planning or problem diagnosis tasks.

• Have you ever asked yourself any of the following?
• Did anything unusual happen with my Queue Manager over the weekend?
• Is queue or channel XYZ still being used? …and, if so, by who/what?
• What type of MQ clients are using my Queue Manager? JMS ? C ? Java ? C++ ?
• What version of MQ clients are connecting to my Queue Manager? Are any of the clients old and need updating?
• What channels are being used and what is the relative split of persistent message traffic?
• How efficient are my channels? What is the average batch size in use over time?
• Are my channel exits efficient? What time is being spent in them?
• How is my network performing? How does my network round-trip time vary over time?
• How does the activity/depth of queue X vary over the day? What are my peak traffic times?
• How does the activity/depth of queue X vary over the last 3 months? Am I seeing an increase of traffic?
• Will my MQ infrastructure be able to cope with future demands?
• Are my channels having to ever retry their puts to deliver messages?
• Are any of my applications misbehaving? Do we see excessive amounts of failed MQI calls such as MQGETs, short connections, or are they opening/closing the queue for each message?
• How is my processing speed varying over time? How long are messages languishing on queues?
• Am I delivering on my Service Level Agreements (SLA) ?
• Are my applications publishing to topics with no subscribers?
• What is the maximum depth my queue has reached during a particular period?

These types of questions, and many more, are the type of questions you can answer if you have a history of your event, accounting and statistics messages. With MQEV all you need to have is a couple of queues on your Queue Manager – no need for an expensive and complicated database.

To view, search and aggregrate the data collected and stored by MQEV, you can use command line commands through MQSCX, or a GUI interface through MO71. To use either of MQSCX or MO71 for MQEV purposes, no further licence is required beyond what is needed for MQEV. Until June 30th 2020, no licence is needed at all.

MO71 GUI displaying MQ Event messages in MQEV

MQSCX command line displaying MQ Event messages in MQEV

The product is free for you to download and try out until June 30th 2020. You don’t even need to register any details to play with it. The product is free to download from our website.

All we ask in return is that in a week or two when you’ve played with it you send us an email telling us what you liked and what you didn’t like about it and perhaps what features you’d like to see in the future.

We look forward to hearing your comments. Please send them to support@mqgem.com.