Running MQEV as an MQ SERVICE

If you have IBM MQ Event Switches enabled, whenever your queue manager is running, you should have a program consuming the event messages that are generated. If you don’t, your event queues will fill up, and newer event messages will be silently discarded by the queue manager.

One way to ensure that there is always a program running, and consuming these messages, while the queue manager is running, is to define that program as an MQ SERVICE object, and let the queue manager start it up. You can do this with any program, and MQEV is no exception.

In MQEV V9.2.0, we added an additional flag, -k, that tells MQEV it is being run in this mode which changes the way it behaves a little:-

• It will run in background mode (just as if you had specified -b)
• It will not retry connection failures, but instead just end. It will be running as a locally bound connection, and so these failures will be as a result of the queue manager ending (normally or abnormally). When the queue manager next starts up, so will MQEV. It was possible to mimic this prior to MQEV V9.2.0 by using -r none, but that had the effect of suppressing retry of all failures. -k only suppresses connection failure retry.

So to run MQEV as an MQ service, you need a definition like this on your queue manager (along with the two queues that MQEV uses).

DEFINE SERVICE(MQGEM.MQEV) +
       DESCR('MQGem Software Event Monitor') +
       CONTROL(QMGR) SERVTYPE(SERVER) +
       STARTCMD('mqev') STARTARG('-m +QMNAME+ -k') +
     [ STOPCMD('mqscx') STOPARG('-m +QMNAME+ -f -C "=mqev;STOP EV"') ]

To explain the various parameters:-

• CONTROL(QMGR) ensures the program is started when the queue manager starts up.
• SERVTYPE(SERVER) says that there is only one instance of this SERVICE running at any one time, and allows you to use the DISPLAY SVSTATUS command to check whether it is running.
  

  MO71 showing a list of Service Status containing MQEV

• The STARTCMD names the program, which in my case is in the path, and STARTARG provides the parameters. The use of +QMNAME+ is a short-hand on SERVICE objects for the name of the owning queue manager. This means you don’t have to change the definition to copy it to other queue managers.
• The STOPCMD is optional, because MQEV will end when it detects the queue manager is quiescing, or gets any other connection failure.
  • It might be useful to code the STOPCMD and STOPARG if you are developing your event scripts, for easy stop and restart of MQEV using the STOP SERVICE and START SERVICE MQSC commands. However, you can just as easily issue the STOP EV command via MQSCX or MO71 interactively.
  • The STOPARG are parameters to the MQSCX program, and tell it to run the initial command (-C) of =mqev followed by STOP EV.

---

If you don’t have a licence and would like to try out MQEV then send an email to support@mqgem.com and a 1-month trial licence will be sent to you.

MQEV on z/OS

The recent release of MQEV V9.2 included the provision of MQEV to run natively on the z/OS platform.

When collecting event messages from your IBM MQ Queue Manager, it is most appropriate to run the event collecting application local to the queue manager. To this end, getting MQEV running natively on z/OS was important to us.

Started Task

You can run MQEV in a number of environments on z/OS, but we imagine running it as a started task will be the most prevalent. Here’s an example started task procedure that you might copy to your procedure library and name something like MQG1MQEV to go along with your queue manager MQG1’s tasks MQG1MSTR and MQG1CHIN. The IBM-supplied procedure library is called SYS1.PROCLIB, but your installation might use its own naming convention.

//*********************************************************************
//* Run MQEV program                                 
//*********************************************************************
//         PROC QMGR=MQG1
//PROCSTEP EXEC PGM=MQEV,
//         PARM=('-m &QMGR')
//*
//STEPLIB  DD DSN=GEMUSER.USER.LOAD,DISP=SHR                         
//         DD DSN=IBM.MQ.SCSQAUTH,DISP=SHR                           
//         DD DSN=IBM.MQ.SCSQANLE,DISP=SHR                           
//SYSPRINT DD SYSOUT=*  
//SYSOUT   DD SYSOUT=*      
//MQGEML   DD DSN=GEMUSER.MQGEM.LIC,DISP=SHR
//MQEVMQX  DD DSN=MQGEM.MQEV.SCRIPTS(MQEV),DISP=SHR
//MQEVLOG  DD DSN=MQGEM.MQEV.LOGS.&QMGR,DISP=SHR
//

There are a number of DD names coded in the above JCL which make setting up your MQEV configuration easier, and keep the JCL PARM string short.

DD name MQGEML

The MQEV program must be able to find your MQGem Software licence file in order to be able to run. This is a licence file specific for MQEV on z/OS. A distributed platform licence will not enable MQEV on z/OS to run. The MQGEML DD name points to that file, which can be an MVS file or an HFS file. All MQGem licences can be concatenated into one file, so it is appropriate to put them all in a central location and have all MQGem Software programs look to the same place.

DD name MQEVMQX

The MQEV program must be able to find the MQEV script. This can be located using the -f program parameter, however in JCL it is neater to use this DD name to locate it instead. The script file can be an MVS file or an HFS file.

DD name MQEVLOG

The MQEV program needs to know where to write out its log files. This can be located using the -L program parameter, however in JCL it is neater to use this DD name to locate the MVS library or HFS directory instead. Note in this example how the library name contains the queue manager name which is very good practice when setting up the logging for MQEV.

Starting and Stopping MQEV

If MQEV is defined as a procedure you can start it using the MVS start command, thus:

COMMAND INPUT ===> /START MQG1MQEV

MQEV can be stopped by sending the STOP EV command to it’s command server (i.e. via the MQGEM.MQEV.COMMAND.QUEUE) using MQSCX or MO71. MQEV on z/OS has an additional stop mechanism as it is coded to listen for the MVS stop command. So you can choose to stop it using the MVS stop command, thus:

COMMAND INPUT ===> /STOP MQG1MQEV

Interactive

When developing MQEV scripts to act upon different events being noticed, you may find it useful to run MQEV interactively. This allows you to debug your scripts by stepping through them line by line, or to set breakpoints to stop on specific situations.

MQEV on z/OS can run interactively in z/OS UNIX, in OMVS or in TSO.

For example, here’s MQEV being run in debugging mode in TSO, inspecting some event variables when the MQEVEvent() function is called.

 FNC:    1 > func MQEVEvent()
 DBG>
print event.summary
 Command - Change Qmgr - Qmgr:MQG1
 DBG>
print event.evobjname
 MQG1
 DBG>

MQEV running on z/OS can process event messages and operate on them just in the same was as MQEV running on a distributed platform and connecting to your z/OS queue manager over a client. Running MQEV on z/OS gives you the benefit of avoiding the cost of the client connection and keeping all your event data processing on the z/OS machine.

Tools to interrogate MQEV to discover the details of event messages can either run on z/OS (for example MQSCX for z/OS) or can attach as clients (for example MO71). In this latter case, answers to questions about event messages are delivered over client connections, but in this case only the data the user is interested in need be sent over the client connection, rather than every single event message.

---

You can download MQEV for z/OS from the MQEV Download Page. If you’d like to try out MQEV on z/OS, please email support@mqgem.com and a 1-month trial licence will be sent to you.

MQEV version 9.2.0 is released

MQGem Software is pleased to announce that a new version of MQEV, our Event Processing tool for IBM MQ, is now available.

The main features of the release are as follows:-

Run MQEV as a Queue Manager SERVICE object

There is now a -k parameter to indicate MQEV is running as an IBM MQ service. This mode of execution essentially says that MQEV should start and end in line with the Queue Manager itself. So, when the Queue Manager ends MQEV will not attempt any retries. Here’s an example of how you might define your MQEV SERVICE object.

DEFINE SERVICE(MQGEM.MQEV) +
       DESCR('MQGem Software Event Monitor') +
       CONTROL(QMGR) SERVTYPE(SERVER) +
       STARTCMD('mqev') STARTARG('-m +QMNAME+ -k')

Read more about this is Running MQEV as an MQ SERVICE.

MQEV now has a STOP EV command

An administrator can end MQEV at any time by issuing the STOP EV command. This can be issued from MQSCX or through MO71.

New and changed expression functions

MQEV uses the MQSCX control language in its scripting, and there are some new and changed expression functions in this new release (and also in the new MQSCX release – see that post for examples).

• New: valueof()
  String function to parse files in the format FIELD(VALUE)
• New: power(x,y)
  Mathematical function returning the value of x^y
• Changed: system()
  The system() function now has an option second parameter to determine whether to run the invoked command synchronously or asynchronously

z/OS Support

MQEV is now available to run locally on the z/OS platform. To enable this you require a z/OS specific licence. A distributed platform MQEV licence will not enable MQEV on z/OS to run. For more on this read this blog post: MQEV on z/OS

Command Level support up to 920

MQEV now supports queue managers up to the most recently released command level of 920.

---

The new version can be downloaded from the MQEV Download Page. Any current licensed users of MQEV can run the new version on their existing licence. If you don’t have a licence and would like to try out MQEV then send an email to support@mqgem.com and a 1-month trial licence will be sent to you.

Using ALERTS with MQEV

Our event, accounting and statistics monitoring program, MQEV has a feature to raise alerts. They were initially designed to allow you to notify an MQ administrator of something untoward that has been discovered through an event message or with accounting and statistics data, but they actually serve a number of other purposes. This blog post is going to look at the various different ways you can make use of MQEV alerts.

• Notification to the MQ Administrator
  This is the first, and perhaps most obvious use of alerts in MQEV. For example, upon receiving a Queue Full event, you might raise an alert to notify the MQ administrator of the problem. This could be accompanied by sending an email or text message, for example. In addition, MQEV publishes an MQ message when an alert is raised so that other programs can subscribe to them for speedy notification.

  ****************************************************************
  * Function for processing an event
  ****************************************************************
  func MQEVEvent()
    @objname = event.evobjname
    @objtype = event.evobjtype
    if (event.evreason = PERQDPFULL)
      ************************************************************
      * Alert the MQ Administrator
      ************************************************************
      ADD EVALERT TEXT('Queue High') CATEGORY(VIP) +
          EVOBJNAME('<@objname>') EVOBJTYPE(<@objtype>) +
          SEVERITY(SEVERE) REPLACE
    endif
  endfunc

• Driving a script after some time passes
  This example is something that is also shown in the MQEV manual. In this case you have noticed a situation that, if it remains for too long, is a problem. However, if it clears up quickly, there is no need to trouble the MQ Administrator. The example uses the receipt of a Queue High event. It raises an informational alert with a short expiry time, and if the Queue High status is still present, and has not been cleared with a subsequent Queue Low event, the expiration of the alert will drive code to notify the MQ administrator of the problem.

  ****************************************************************
  * Function for processing an event
  ****************************************************************
  func MQEVEvent()
    @objname = event.evobjname
    @objtype = event.evobjtype
    if (event.evreason = PERQDPHI)
      ************************************************************
      * Add a temporary alert that will expire in 60 seconds
      ************************************************************
      ADD EVALERT TEXT('Queue High') CATEGORY(QHIGHTEMP) +
          EVOBJNAME('<@objname>') EVOBJTYPE(<@objtype>) +
          SEVERITY(INFO) RETINTVL(60) REPLACE
    endif
    if (event.evreason = PERQDPLO)
      ************************************************************
      * Remove both alert types, temp & final, wildcarded CATEGORY
      ************************************************************
      REMOVE EVALERT TEXT('Queue High') CATEGORY(QHIGH*) +
             EVOBJNAME('<@objname>') EVOBJTYPE(<@objtype>)
    endif
  endfunc
  ****************************************************************
  * Function called when an alert expires
  ****************************************************************
  func MQEVAlertExpire()
    if (alert.category = 'QHIGHTEMP')
      @objname = alert.evobjname
      @objtype = alert.evobjtype
      ADD EVALERT TEXT('Queue High') CATEGORY(QHIGH) +
          EVOBJNAME('<@objname>') EVOBJTYPE(<@objtype>) +
          SEVERITY(WARN) REPLACE
    endif
  endfunc

• Drive Code repeatedly
  An extension of the above idea, is to use expiring alerts to drive code on a regular basis to check on something, perhaps a behaviour in MQ that isn’t able to be monitored by event messages. After checking the attribute, a new expiring alert, based on the one that just expired, is created, and round it goes again. The MonitorFunction() might also create alerts of the first type.

  ****************************************************************
  * Function: addEVAlert
  ****************************************************************
  func addEVAlert(Text, Category, RetIntvl)
    =mqev
    ADD EVALERT TEXT('<@Text>') CATEGORY('<@Category>') +
        RETINTVL(<@RetIntvl>) EVQMGR(<_qmgr>) +
        SEVERITY(INFO) REPLACE
    =mqsc
  endfunc
  ****************************************************************
  * Function called when we connect
  ****************************************************************
  func MQEVConnected()
    **************************************************************
    * Initialise the expiring alerts that drive the monitoring fns
    **************************************************************
    addEVAlert("Delay: 2 Hours",   "DELAY.2HRS",  2 * 60 * 60)
  endfunc
  ****************************************************************
  * Function called when we an alert expires
  ****************************************************************
  func MQEVAlertExpire()
    if (alert.CATEGORY = 'DELAY.2HRS')
      MonitorSomething(...)
      * Now remake the same alert *
      addEVAlert(alert.TEXT, alert.CATEGORY, alert.RETINTVL)
    endif
  endfunc

• Drive some code in an ad hoc manner
  Again, in an extension to the above model, you could have code to run that is driven by an expiring alert, and could be driven in an ad hoc manner by an administrative command creating an alert with a very short expiry.

  ****************************************************************
  * Function called when we an alert expires
  ****************************************************************
  func MQEVAlertExpire()
    if (alert.CATEGORY = 'DRIVE.NOW')
      CheckSomething(...)
    endif
  endfunc

  This code could be driven by an MQ administrator using the following command:

  ADD EVALERT TEXT('Drive code') CATEGORY(DRIVE.NOW) +
      RETINTVL(1) SEVERITY(INFO) REPLACE

• State Variable
  When monitoring for situations using event messages or even issuing direct MQ commands, you can raise an alert for an MQ Administrator to take notice of what has been discovered. If the same situation is seen again, and the MQ Administrator has not yet noticed the previous alert, you may decide that you don’t need to contact him again. The existence of the alert in the system marks that he has already been notified. Once the MQ Administrator deals with the alert, he can delete it. The lack of an alert in the system would then mean that the script would notify him again if the situation subsequently arose.

  ****************************************************************
  * Function: AlertUser 
  * Purpose: This function will alert the user in whichever way is
  *          needed, for example, raise an alert, or send an email
  ****************************************************************
  func AlertUser(objname, objtype, category, alert_text)
    =mqev
    DIS EVALERT CATEGORY('<@category>') +
        EVOBJTYPE(<@objtype>) EVOBJNAME('<@objname>') +
        EVQMGR(<_qmgr>)
    if (_matches = 0)
      ************************************************************
      * Send email to notify MQ Administrator *
      ************************************************************
      system('email.bat "' + @alert_text + '"')
    else
      @id = EVALERT
      REMOVE EVALERT(<@id>)
    endif
    **************************************************************
    * Always add a new replacing alert so that the date, and any
    * new text in the alert, is updated
    **************************************************************
    ADD EVALERT TEXT('<@alert_text>') CATEGORY('<@category>') +
        EVOBJTYPE(<@objtype>) EVOBJNAME('<@objname>') +
        EVQMGR(<_qmgr>)
    =mqsc
  endfunc

As you can see, MQEV and its alerts are very flexible. Alerts can notify an MQ Administrator of an event within your MQ estate, and in addition can be used to control when those alerts might be raised.

---

The MQEV product is free for you to download and try out until June 30th 2020. You don’t even need to register any details to play with it. You can download it from our website.

All we ask in return is that in a week or two when you’ve played with it you send us an email telling us what you liked and what you didn’t like about it and perhaps what features you’d like to see in the future.

We look forward to hearing your comments. Please send them to support@mqgem.com.

MQEV Topologies

MQSCX and MO71 provide admin for MQEV

MQEV uses a couple of queues in order to operate. A command queue (CQ) to receive inquiries from tools such as MO71 and MQSCX; and a persistence queue (PQ) where compressed event, accounting and statistics data is held once received from the IBM MQ event queues (EQ).

The program itself can run using a client or local bindings connection. This provides a number of choices for the topology arrangement that you can use with MQEV. In this post we discuss our two main recommended topologies.

All queues local to MQEV

This is the ideal approach. It uses a local connection for access to all the queues involved.

In this topology, MQEV runs as a locally bound application on the same machine as the queue manager being monitored. The event queues (EQ) and the persistence queue (PQ) are hosted on the same queue manager. Commands are sent to MQEV via the command queue (CQ), by administrative tools such as MQSCX and MO71. These tools may be connected either locally or over a client connection, just as they do for the IBM MQ Command Server.

In the diagram below, the MO71 administrative tool is shown in the centre of the diagram on a separate machine to the queue managers, but of course it is possible that the machine where MO71 is running may also host one of the monitored queue managers.

To run the MQEV program in this way, issue a command of the following format (where MQG1 is the monitored queue manager):

mqev -m MQG1

MQEV Topology: MQEV runs local to each queue manager to be monitored

Separate state QMgr for the persistence queues

There may be times when running MQEV local to the monitored queue manager is not appropriate, or even possible. For example, MQEV clearly can’t run locally on the MQ Appliance. Additionally, installations may have rules about what programs can run local to a queue manager.

In such cases, MQEV can connect as a client to the monitored queue manager, but maintain its persistent state on a local queue manager, ideally one dedicated to run your MQEV instances. This state queue manager holds the persistence queues (PQ), one for each monitored queue manager, and the event queues (EQ) are read by MQEV over a client connection. Commands are sent to MQEV via the command queue hosted on each monitored queue manager as before, also read by MQEV over the client connection. Administrative tools such as MQSCX and MO71 connect to the queue managers just as they do for the IBM MQ command server. They do not connect to the MQEV state queue manager.

To run the MQEV program in this way, issue a command of the following format (where STATEQM is the state queue manager, and -l indicates the connection to MQG1 is over a client):

mqev -m MQG1 -l -s STATEQM

The client connection in this case should be configured with the use of a CCDT file, by defining a CLNTCONN channel definition with a QMNAME of MQG1.

MQEV Topology: Dedicated state queue manager for MQEV persistence queues

If you have other topology requirements that are not covered by the above two setups, we would be interested to hear from you.

---

The MQEV product is free for you to download and try out until June 30th 2020. You don’t even need to register any details to play with it. You can download it from our website.

All we ask in return is that in a week or two when you’ve played with it you send us an email telling us what you liked and what you didn’t like about it and perhaps what features you’d like to see in the future.

We look forward to hearing your comments. Please send them to support@mqgem.com.

First Steps with MQEV and MQ Accounting and Statistics

Our newest product MQEV, can be used to collect, process, store and search the data generated by your IBM MQ Queue Manager as Accounting and Statistics messages. In this post, we will take you through the first steps of using MQEV to look at Accounting and Statistics messages.

We assume that you have downloaded the MQEV program from our website, and also grabbed the latest version of either MQSCX or MO71 to interact with MQEV. You don’t need to have a licence file for either of these products in order to use them with MQEV – they are covered by the MQEV licence, and will work for free without any licence until June 30th 2020. Installation of MQGem products is really easy – just unzip/untar into a location that is in your path and off you go.

Make sure your queue manager is running and then run the provided script file config.mqx to define the two queues that the MQEV product requires in order to work (MQGEM.MQEV.COMMAND.QUEUE and MQGEM.MQEV.DATA.QUEUE). You may have already done this step if you have used MQEV with MQ Event messages.

Now run the MQEV program with the following command (assuming your queue manager is called MQG1).

mqev -m MQG1

Now MQEV is running, but it’s not capturing any accounting and statistics yet – first you have to tell it what you’re interested in. First turn them on in your queue manager with the following MQSC commands (or their equivalent action in your favourite GUI). I’m just enabling MQI statistics, and Queue accounting for my queue named Q1. I’ve shortened the Statistics Interval a bit too so you don’t have to wait as long to see some data. You can turn on Queue accounting (and queue statistics) for all queues at once using the queue manager attribute of the same name, but for now, I’m just looking at a single queue. There are in fact five different types of accounting and statistics for you to use. Read more about them here.

ALTER QMGR STATMQI(ON) STATINT(900)

ALTER QLOCAL(Q1) ACCTQ(ON)

Now your queue manager will generate some accounting and statistics, and the next step is to tell MQEV to listen out for them. You can do this using a command line tool such as MQSCX (in =mqev mode) or using the MO71 GUI. Open one of the twisties below to see instructions.

Command Line: MQSCX

Issue the following commands:

=mqev
RESUME EVQ(SYSTEM.ADMIN.ACCOUNTING.QUEUE)
RESUME EVQ(SYSTEM.ADMIN.STATISTICS.QUEUE)

If you now issue the following command your will see that the two queues you resumed above, now show STATUS(ACTIVE).

DISPLAY EVQ(*)

Using MO71 GUI

Right click on the queue manager in the main window and from the context menu choose MQ EV Events -> MQ Event Queue List …
Press the “Refresh” button to populate the list.
You will be presented with a list of the known event queue names (these are pre-configured to MQEV for you, but all in a suspended state so that you control when to start reading from them). Select the following two queues, the right click and choose “Resume” from the context menu.

• SYSTEM.ADMIN.ACCOUNTING.QUEUE
• SYSTEM.ADMIN.STATISTICS.QUEUE

Resume context menu on a list of MQEV event queues

You will see the dialog list automatically refresh and show that your two queues now have a status of “Active” instead of “Suspended”.

Now you are going to generate some accounting data by connecting an application that makes use of your queue. You can use any MQ application you like, but for simplicity, I’m just using the IBM-supplied sample amqsput.

amqsput Q1 MQG1

Accounting information is written out by the IBM MQ Queue Manager when an application disconnects. So once you have finished putting some messages using an MQ application, make sure to end the application so that it disconnects, and then you will be able to see the data. Take a look at accounting data now. Open one of the twisties below to see instructions.

Command Line: MQSCX

Issue the following commands:

=mqev
DISPLAY ACCTQ(*)

You will see output something like this:-

[09:30:45] =mqev

Commands directed to MQEV

[09:30:53] DISPLAY ACCTQ(*)

ACCTQ(Q1) INTVLEND(2020-03-09 09:30:44 (Local)) ALLPUT(8)

Total display responses – Received:1

MQEV:MQG1>

Using MO71 GUI

Right click on the queue manager in the main window and from the context menu choose MQEV Acct & Stat -> MQ Accounting Queue List …
Press the “Refresh” button to populate the list.
You’ll see one line in the list like the screenshot below.

MO71 showing accounting data captured by MQEV

This just shows a summary of the accounting data. Now let’s look at all the details in this record.

Command Line: MQSCX

Repeat your last command and add ‘ALL’ to the end to see the full data in the record.

DISPLAY ACCTQ(*) ALL

You’ll see output like the following.

[09:31:51] DISPLAY ACCTQ(*) ALL

ACCTQ(Q1) QTYPE(QLOCAL) DEFTYPE(PREDEFINED)

APPLNAME(E:\mqm9130\bin64\amqsput.exe) USERID(mqgemusr)

CONNID(414D51434D51473120202020202020207E20475E01663522) PID(87308)

TID(1) EVSTREAM($ACCTQ) EVQMGR(MQG1)

INTVLSTA(2020-03-09 09:30:43 (Local)) INTVLEND(2020-03-09 09:30:44 (Local))

OPENTI(2020-03-08 20:30:40 (UTC)) CLOSETI(2020-03-08 20:30:44 (UTC))

RECORDS(1) OPENCNT(1) CLOSECNT(1) ALLPUT(8)

PUT(8) PUTNP(8) PUTBYTE(24) PUTBYTENP(24)

PUTMINBYTENP(3) PUTMAXBYTE(3) PUTMAXBYTENP(3)

Total display responses – Received:1

MQEV:MQG1>

In this output you can see that the application opened the queue once (OPENCNT), and put 8 non-persistent messages(PUTNP), with a total of 24 bytes of data (PUTBYTE). You’re also shown identifying fields about the application, its name, the user id that ran it, the Connection ID and process and thread IDs.

Using MO71 GUI

Double click on the record in the list dialog to take a look at the details.

MO71 display of an individual accounting record

On the ‘General’ tab you’ll see information identifying the application, its name, the user id that ran it, the process and thread ID and the Connection ID. You’l also see that it opened the queue once.

On the ‘Messaging’ tab you’ll see the number of puts it did, in my example there were 8 non-persistent puts.

On the ‘Bytes’ tab you’ll see how many bytes of message data it put in total.

By now, your Statistics Interval should have passed, and there will be some Statistics data to look at too. Statistics data is recorded by the IBM MQ Queue Manager on a regular interval, as defined by the STATINT queue manager attribute. These statistics are for all the MQI calls made across the whole queue manager, not just your one application using queue Q1. Open one of the twisties below to see instructions.

Command Line: MQSCX

Issue the following command:

=mqev
DISPLAY STATMQI(*)

You will see output something like this:-

[10:22:22] DISPLAY STATMQI(*)

STATMQI($STATMQI) INTVLEND(2020-03-09 10:22:17 (Local)) ALLPUT(392)

GET(279)

Total display responses – Received:1

MQEV:MQG1>

Repeat the command with ‘ALL’on the end to see even more details.

Using MO71 GUI

Right click on the queue manager in the main window and from the context menu choose MQEV Acct & Stat -> MQ Statistics MQI List …
Press the “Refresh” button to populate the list.
You’ll see one line in the list like the screenshot below.

MO71 showing statistics data captured by MQEV

Double click on the line, to see the full details of the record.

There are five different types of Accounting and Statistics data written by an IBM MQ Queue Manager, and there is an MQEV command for displaying each type.

Command Line: MQSCX	Using MO71 GUI
DISPLAY ACCTMQI(*) DISPLAY ACCTQ(*) DISPLAY STATCHL(*) DISPLAY STATMQI(*) DISPLAY STATQ(*)

By enabling Accounting and Statistics on your queue manager, MQEV can collect and store the data for your queues, and you can display the activity at any time, if necessary going back several days or weeks. You can of course, even graph the activity to more easily show how it changes over time. Watch this space for more on graphing.

Try out enabling some of the other types of IBM MQ Accounting and Statistics, collecting them with MQEV (both queues that you need to read from have been resumed in MQEV by following this post), and displaying the data you receive.

Look out for more posts about how you can manipulate the different views of the data that Accounting and Statistics gives you.

---

The MQEV product is free for you to download and try out until June 30th 2020. You don’t even need to register any details to play with it. You can download it from our website.

All we ask in return is that in a week or two when you’ve played with it you send us an email telling us what you liked and what you didn’t like about it and perhaps what features you’d like to see in the future.

We look forward to hearing your comments. Please send them to support@mqgem.com.

Act on MQ Event messages

Some event messages tell you something that you might like to act upon immediately. Others you might just want to keep an eye on, for example channel problems where channel retry will probably cover it.

In this blog post, I’ll demonstrate an example of how you might use MQEV to immediately act upon an event message. In our demonstration today, we have a cluster of queue managers with a number of hosted queues called CLUS.Q1. Using a similar tactic to that used by the The Cluster Queue Monitoring sample program (AMQSCLM), we’re going to lower the CLWLPRTY value of a cluster queue when it becomes a less useful choice, in this case when it hits the Queue-High state.

Turning on Queue Depth Events in IBM MQ

On each cluster queue manager, we would turn on Queue Depth events with the following commands, and set an initial value of CLWLPRTY to allow it to be lowered.

ALTER QMGR PERFMEV(ENABLED)

ALTER QLOCAL(CLUS.Q1) MAXDEPTH(10000) +
      QDEPTHHI(40) QDEPTHLO(10) +
      QDPHIEV(ENABLED) +
      CLWLPRTY(5)

Explanation of Queue Depth settings

In this example, rather than following the usual Queue Depth High of 80% and Queue Depth Low of 20%, with, say a maximum depth of 5000, I’ve doubled the maximum depth to 10000, and set the percentages so I still get a Queue High at 4000, and Queue Low at 1000. This gives me much more head-room before I hit Queue Full, but with 4000 messages on the queue, that’s still a hefty backlog to work through.

Turning on Queue Depth Events in MQEV

To ensure MQEV is processing performance events, issue the following command to MQEV.

RESUME EVQ(SYSTEM.ADMIN.PERFM.EVENT)

Processing Queue Depth Events in MQEV

Now we’re going to add some instructions to MQEV, to tell it what do to when it receives Queue Depth events. This is done by using the MQEVEvent() function in the mqev.mqx script. In the script below it tests for the event type, looking for performance events (event.evtype = PERFM), and then displays the queue that the event was all about, since we need to tell whether it was a cluster queue, and if so what it currently has as a value for CLWLPRTY.

*********************************************************************
* Function for processing an event                                  *
*********************************************************************
func MQEVEvent()
  if (event.evtype = PERFM)
    =mqsc
    @qname = event.evoobjname
    DISPLAY QLOCAL(<@qname>) ALL
    if (CLUSTER OR CLUSNL)
      ...
    endif
  endif
  endfunc

If it finds that this is indeed a cluster queue, then it will do some adjusting of the CLWLPRTY value based on whether this is Queue High, Queue Low, or Queue Full.

If you have a good naming convention for your queues, you could avoid the step of displaying the queue, and rely on your naming convention pattern to inform you that it is a cluster queue.

*********************************************************************
* Function for processing an event                                  *
*********************************************************************
func MQEVEvent()
  if (event.evtype = PERFM)
    =mqsc
    @qname = event.evoobjname
    DISPLAY QLOCAL(<@qname>) ALL
    if (CLUSTER OR CLUSNL)
      if (event.evreason = PERQDPHI)
        ALTER QLOCAL(<@qname>) CLWLPRTY(4)
      endif
      if (event.evreason = PERQDPFULL)
        ALTER QLOCAL(<@qname>) PUT(DISABLED)
      endif
      if (event.evreason = PERQDPLO)
        ALTER QLOCAL(<@qname>) CLWLPRTY(5) PUT(ENABLED)
      endif
    endif
  endif
endfunc

When it hits a Queue High state, it lowers the CLWLPRTY by one, making it a less good choice than instances of the queue on other queue managers. Of course, if all instances of the queue are lowered in the same way, they all become equally appropriate choices again. You might, in addition, want to raise an alert to ensure the MQ System administrator is aware of the situation.

When it hits a Queue Full state, it changes the queue to be PUT(DISABLED). This is because that attribute is one of the ones flowed round the cluster, and it will stop this queue from being chosen as a result.

When it hits a Queue Low state, it sets the CLWLPRTY back to 5 and re-enables the queue for putting again.

These are just example actions to show you how you could act upon an event message at the moment it arrives on the event queue. You can of course, issue any command you like. You could for example, start more instances of your application, perhaps by running a script using the MQSCX system() function, or by defining it as an MQ SERVICE object and starting that.

---

The MQEV product is free for you to download and try out until June 30th 2020. You don’t even need to register any details to play with it. You can download it from our website.

All we ask in return is that in a week or two when you’ve played with it you send us an email telling us what you liked and what you didn’t like about it and perhaps what features you’d like to see in the future.

We look forward to hearing your comments. Please send them to support@mqgem.com.

Does MO71 log the commands it issues?

This is a question that we get asked quite regularly, so we thought it worth writing up the answer as a blog post.

If you’re asking this question, there are two reasons that immediately come to mind that might be behind your question:-

• You just want some sort of documentary log, a history of what you have done to look back on.
• You need a log of commands issued for audit reasons.

However, applications logging their own activity have a number of disadvantages. For example, if the log file were on your own machine, then you have to manage those files, house-keep them as they get old enough to delete, etc. Flat log files are not very easy to search through, say to find out what happened last weekend, or all the activity associated with a particular object.

If you want something for audit reasons, using a log file on the user machine has a number of additional problems:-

• Firstly, it is not audit proof because
  • It is easy to circumvent
  • It is incomplete since other tools might be used to issue commands to the queue manager
  • The user could switch off the logging feature in their tool, or install a new version of it to avoid the logging configuration
• Secondly, it is hard to collect the logs from all the different users’ machines to have the complete picture.
• Thirdly, if individual applications are creating the log file, e.g MO71, runmqsc, MQ Explorer and so on, then you get a myriad of different log file formats.

To solve all of the above, IBM MQ Queue Managers have a feature called Command Events. When enabled these result in an event message being emitted by the queue manager for any command issued (with a tweak to avoid DISPLAY commands if you wish – see EVENT Switches). This is a far better solution than logging in the user-facing tool (e.g. MO71), for reasons such as:-

• Central collection is better because it is easier to obtain a complete record, since the records are not segregated across many user machines
• It cannot be subverted by non-privileged users
• The format is consistent regardless of the tool used by the user
• No updates are required to any tool, the logging capability is available immediately for all applications

If all you want is something to pass an audit, command and configuration events might be enough. But if you want a record for diagnostic purposes there are other things you might wish to add to the record of what happened, such as:-

• Channel problems via Channel events
• Queues High or Full via Performance events
• Queue Service Interval High via Performance events
• Queue Manager Fail-overs via Start/Stop events

We have been recommending using Command (and Configuration) events rather than logging at the user-side MQ administration tool, for many years. Previously the problem with this recommendation was the collection and searching of the event data. However, we have recently released a new product, MQEV, which will help you in this regard. MQEV will collect, process and store your event data, and make searching and report building very simple. You can now easily get the information from the event messages that your IBM MQ queue manager is designed to emit. If required you can process the events real-time, and make changes immediately, or even raise alerts to your MQ administrators for follow-up.

---

Lumberjack image courtesy of vectorolie at FreeDigitalPhotos.net

Configuration Change Events in MQEV

When a change is made to your queue manager’s configuration, you can be delivered event messages notifying you of the change. These event messages are called Configuration Events.

When the notification being made is about a change in the configuration (rather than the creation or deletion of an object), then you are delivered two event messages; a before image and an after image of the object being changed. In order to determine what actually changed, the two event messages must be compared to find the differences. Both events, the before and the after, contain all the attributes of the object. To find what changed, all attributes in each event must be compared. In the screenshot below, two browse windows showing the two event messages are shown side by side.

Manually Comparing two Config Events

Using MQEV to display your event messages instead of simply browsing them as formatted PCF messages can improve your understanding of what happened hugely.

The default display output from the MQEV command server will include a summary of the change event.

SUMMARY(Config - Change Object - Queue:Q1 - MAXDEPTH[5000 -> 10000])

MQEV has compared the two event messages that were emitted by the queue manager, and discovered what the change was and provided a very helpful summary.

You can, of course, look at the whole event message if you need to. For example in MQSCX, you can use the following command to show all the attributes of the object. MQEV saves on storage space by only storing one copy of the attributes that haven’t changed. Essentially, the two configuration events are combined inside MQEV into one item.

DISPLAY EVENTS(*) EVTYPE(CONFIG) EVOBJNAME(Q1) ALL

MQSCX Extended MQSC Program – Version 9.1.0

Licenced to Paul Clarke

Licence Location: Home Office

[16:44:32] DISPLAY EVENTS(*) EVTYPE(CONFIG) EVOBJNAME(Q1) ALL

EVQMGR(MQG1) EVENTS($EVENTS) EVTIME(2020-02-07 11:55:35 (Local))

EVREASON(CFGCHGOBJ) EVTYPE(CONFIG) EVUSERID(mqgemusr) EVOBJNAME(Q1)

EVOBJTYPE(QUEUE) EVENTID(00000286) CFHCMD(43) CFHREASON(2368)

SUMMARY(Config – Change Object – Queue:Q1 – MAXDEPTH[5000 -> 10000])

BEFORE:

EVENTUSER(mqgemusr)

EVSID(1D010105000000000005150000001AFA5FFE70975006C3A1C633E903000000000000000000000000)

EVORIGIN(MSG)

EVACCTTK(160105150000001AFA5FFE70975006C3A1C633E903000000000000000000000B)

EVAPPLID( ) EVAPPLTYPE(WINDOWSNT)

EVAPPLNAME(d:\nttools\mqscx.exe) EVAPPLORIG( ) OBJTYPE(QUEUE)

QUEUE(Q1) DESCR( ) PROCESS( ) BOQNAME( )

INITQ( ) TRIGDATA( ) CLUSCHL( ) CUSTOM( )

CLUSTER( ) CLUSNL( ) CRDATE(2019-06-19) CRTIME(06.29.19)

ALTDATE(2019-12-27) ALTTIME(09.24.54) GET(ENABLED) PUT(ENABLED)

DEFPRTY(0) DEFPSIST(NO) MAXDEPTH(5000) MAXMSGL(4194304)

BOTHRESH(0) SHARE(1) DEFSOPT(SHARED) HARDENBO(1)

MSGDLVSQ(PRIORITY) RETINTVL(999999999) USAGE(NORMAL) TRIGCTL(NOTRIGGER)

TRIGTYPE(FIRST) TRIGDEPTH(1) TRIGMPRI(0) QDEPTHHI(80)

QDEPTHLO(20) QDPMAXEV(ENABLED) QDPHIEV(DISABLED) QDPLOEV(DISABLED)

QSVCINT(999999999) QSVCIEV(DISABLED) DISTL(NO) NPMCLASS(NORMAL)

STATQ(ON) ACCTQ(ON) MONQ(HIGH) SCOPE(QMGR)

DEFBIND(OPEN) CLWLRANK(0) CLWLPRTY(0) CLWLUSEQ(QMGR)

DEFPRESP(SYNC) DEFREADA(NO) PROPCTL(COMPAT) IMGRCOVQ(QMGR)

DEFTYPE(PREDEFINED) QTYPE(QLOCAL)

AFTER:

ALTDATE(2020-02-06) ALTTIME(22.55.35) MAXDEPTH(10000)

Total display responses – Received:1

MQEV:MQG1>

If you don’t want to see all the attributes, you can add DISTYPE(CONDENSE) to the end of your command in MQSCX, to get output like the following, where the BEFORE: section just shows the attributes that are changed.

[16:46:37] DISPLAY EVENTS(*) EVTYPE(CONFIG) EVOBJNAME(Q1) ALL DISTYPE(CONDENSE)

EVQMGR(MQG1) EVENTS($EVENTS) EVTIME(2020-02-07 11:55:35 (Local))

EVREASON(CFGCHGOBJ) EVTYPE(CONFIG) EVUSERID(mqgemusr) EVOBJNAME(Q1)

EVOBJTYPE(QUEUE) EVENTID(00000286) CFHCMD(43) CFHREASON(2368)

SUMMARY(Config – Change Object – Queue:Q1 – MAXDEPTH[5000 -> 10000])

BEFORE:

MAXDEPTH(5000)

AFTER:

MAXDEPTH(10000)

Total display responses – Received:1

MQEV:MQG1>

If you’re using MO71 to view events, you’ll notice that MO71 has a new type of dialog designed especially for these change events. Where you might well be used to seeing the dialog show the labels on the left hand side and the values to the right; now the right hand side has two columns. The left shows all the attributes, and the right shows the changes, with a background highlight colour (in case the change made was blanking out a field). You can control this highlight colour from your View->Set Colours.. menu in the usual manner.

MO71 dialog showing a change event

So, now you know what we mean when we say MQEV consolidates, and summarises change events. MQEV doesn’t just capture them and let you look at them later, it helps you to understand more easily what the events are telling you.

---

MQEV is free for you to download and try out until June 30th 2020. You don’t even need to register any details to play with it. You can download it from our website.

All we ask in return is that in a week or two when you’ve played with it you send us an email telling us what you liked and what you didn’t like about it and perhaps what features you’d like to see in the future.

We look forward to hearing your comments. Please send them to support@mqgem.com.

Retaining Events with MQEV

How long should you keep event messages for?

That’s one of those “It depends” questions.

• Is it worth keeping command events for DISPLAY commands at all?
• Is it worth keeping channel retrying events once the channel is up and running again?
• Would you keep “not authorized” events for longer than logger events?

MQEV stores your IBM MQ event data on streams. By default all events are stored on the $EVENTS stream which has a retention interval of 90 days. Streams are your control point for how long to hold onto this data. You might decide that authority events should be kept for much longer, but that channel events can be kept for a shorter time. In which case you might create some new streams thus:-

DEFINE EVSTREAM(CHANNEL) TYPE(EVENTS) RETINTVL(30)
DEFINE EVSTREAM(SECURITY) TYPE(EVENTS) RETINTVL(120)

And then send different event types to their appropriate streams with the following in your MQEVEvent() function in the mqev.mqx script.

*********************************************************************
* Function for processing an event                                  *
*********************************************************************
func MQEVEvent()
  if (event.evtype = AUTHOR)
    _stream = "SECURITY"
  endif;
  if (event.evtype = CHANNEL)
    _stream = "CHANNEL"
  endif;
endfunc

You can still display all your events across these different streams with the usual DISPLAY EVENTS(*) command or the MO71 events list dialog, but if you need to, you can now also filter them by the stream that they are stored in.

MQEV will take care of deleting these events once they pass the retention interval defined on the stream they are stored in.

Of course, this is a very simplistic example. You could separate events out further, for example, by a specific reason. Some channel events might be worth keeping with the authority events, for example “Channel is Blocked” events. Let’s make a small change to the MQEVEvent() function to achieve that.

*********************************************************************
* Function for processing an event                                  *
*********************************************************************
func MQEVEvent()
  if (event.evtype = AUTHOR)
    _stream = "SECURITY"
  endif;
  if (event.evtype = CHANNEL)
    if (event.cfhreason = const.MQRC_CHANNEL_BLOCKED)
      _stream = "SECURITY"
    else
      _stream = "CHANNEL"
    endif;
  endif;
endfunc

And you are not limited there – you can use any data in the event message to determine which stream it should be stored on. For example, you might have a very important application whose retention interval is much higher than everything else, and so make use of the object name in the event, in order to save it off to the appropriate stream. This is where having a good naming convention helps greatly.