Logging Client Connections

A recent discussion about logging of client connections, both those that are blocked and those that succeed, got me thinking that you can capture all the data you need to create such a log with a combination of MQ Event messages and MQ Accounting records (on distributed queue managers). This means you can do this with MQEV. We’ve written about writing log files with the MQSCX control language before, but I thought it would be good to have an MQEV example of this too.

CHLAUTH blocking connections

The IBM MQ Queue manager emits an event message whenever it denies access to a channel connection as a result of matching a blocking CHLAUTH rule. This event message contains a number of very useful pieces of information to identify the channel that was blocked. In the case of a client channel connection these are:-

Field Description MQ PCF Identifier MQEV script response variable
Channel Name MQCACH_CHANNEL_NAME event.CHANNEL
Connection Name MQCACH_CONNECTION_NAME event.CONNAME
SSL Peer Name MQCACH_SSL_PEER_NAME event.SSLPEER
SSL Issuer Name MQCA_SSL_CERT_ISSUER_NAME event.SSLCERTI
Client User Identifier MQCACH_CLIENT_USER_ID event.CLNTUSER
Application Type MQIA_APPL_TYPE event.APPLTYPE
Application Name MQCACF_APPL_NAME event.APPLNAME

Accounting data recording all connections

If you enable accounting data collection, especially ACCTMQI, an accounting record is emitted by the queue manager each time a successful connection disconnects, and also periodically if connections are longer lived than the accounting interval. This record contains a number of very useful pieces of information to identify the application that connected. In the case of a client application these are:-

Field Description MQ PCF Identifier MQEV script response variable
Application Name MQCACF_APPL_NAME data.ACCTMQI
User Identifier MQCACF_USER_IDENTIFIER data.USERID
Connection Name MQCACH_CONNECTION_NAME data.CONNAME
Channel Name MQCACH_CHANNEL_NAME data.CHANNEL
Remote Product MQCACH_REMOTE_PRODUCT data.RPRODUCT
Remote Version MQCACH_REMOTE_VERSION data.RCVERSION

With CHLAUTH and ACCTMQI enabled, you can direct MQEV to collect these emitted records and write the data out to a log file with the format of your choosing.

CONNECT 22:50:53 Channel: MQGEM.SVRCONN from '127.0.0.1' for Application: ':\mqm9300\bin64\amqsputc.exe'
CONNECT 22:54:57 Channel: MQGEM.SVRCONN from '127.0.0.1' for Application: ':\mqm9300\bin64\amqsputc.exe'
BLOCKED 22:55:01 Channel: MQGEM.SVRCONN.TLS from 'GEMWIN10 (127.0.0.1)' for Application: 'MQGem Software MO71'

client_log_2023_05_25.txt

In our example we create a new log file every day by creating the file name using the built-in date formatting function, and then each line we write in the file has a timestamp. This version of the function creates a line in the file suitable for a human to read, but also with eye-catching strings at the start of each line that could be used to sort the file (see example above). An alternative could be to write a CSV file instead of a text file and make the output a comma-separated format instead of just a text string and then you could import it into a spreadsheet later.

Here is an example of how that could look in your mqev.mqx script. First we show you the small function to write out the line to the file.

*********************************************************************
* Function to log client channel connections (failed or successful)
* Allow = 0 - blocked
* Allow = 1 - warning of blocked, thus allowed
* Allow = 2 - connection made
*********************************************************************
func logClient(ChlName, IPAddr, ApplName, Allow, Time)
  @filename = "C:\logs\client_log_" + date(_time, "y_mm_d") + ".txt"

  * Open the logging file in append mode
  @hf = fopen(@filename, "a")
  if (@hf = -1)
    print "Error opening", @filename, "ErroNo=", _errno, _errnostr
    return
  endif

  * Build up the string to write to the logging file
  @logtime    = date(Time, "H:M:S")
  @chldetails = "Channel: " + @ChlName + " from '" + @IPAddr +
              + "' for Application: '" + @ApplName + "'"
  if (@Allow = 2)
    fprint @hf, "CONNECT", @logtime, @chldetails
  endif
  if (@Allow = 1)
    fprint @hf, "WARNING", @logtime, @chldetails
  endif
  if (@Allow = 0)
    fprint @hf, "BLOCKED", @logtime, @chldetails
  endif

  fclose(@hf)
endfunc

Having written our logging function, we can call this whenever we catch an appropriate event message, passing in the time the event happened as the time that should be written in the log file:

*********************************************************************
* Function for processing an event                                  *
*********************************************************************
func MQEVEvent()
  if (event.EVTYPE = CHANNEL)
    ************************************
    * Really blocked the connection    *
    ************************************
    if(event.EVREASON = CHLBLKNOAC OR +
       event.EVREASON = CHLBLKADDR OR +
       event.EVREASON = CHLBLKUSER)
      @Allow = 0
    endif
    ************************************
    * Just warned about the connection *
    ************************************
    if(event.EVREASON = CHLBLKWNOAC OR +
       event.EVREASON = CHLBLKWADDR OR +
       event.EVREASON = CHLBLKWUSER)
      @Allow = 1
    endif
    ************************************
    * Only looking for client channels *
    ************************************
    if (event.CLNTUSER != ' ')
      logClient(event.CHANNEL, event.CONNAME, event.APPLNAME, +
                @Allow, event.EVTIME)
    endif
  endif
endfunc

And we can also call this whenever we catch an appropriate accounting record, weeding out the MCA channels (application names of runmqchl and amqrmppa, and only logging it on the first record (data.SEQNUM = 0) for a long-lived connection, and passing in the connection time (rather than the time the accounting record was created which is much later) as the time that should be written in the log file:

*********************************************************************
* Function for processing an Accounting MQI message                 *
*********************************************************************
func MQEVAcctMQI()
  if (data.CHANNEL AND +
      findstri(data.ACCTMQI, "runmqchl") = 0 AND +
      findstri(data.ACCTMQI, "amqrmmpa") = 0 AND +
      data.SEQNUM = 0)
    logClient(data.CHANNEL, data.CONNAME, data.ACCTMQI, 2, +
              data.CONNTI)
  endif
endfunc

There is, of course, lots more you could do with this. The format of the output file is entirely up to you. If there are other things which are important to log, you can add those. The data available in each type of record that the queue manager emits is shown in the tables at the beginning of the post.

This example has specifically focused on logging client connections, but you could extend this to MCA channels as well by removing the code that ensures only client connections are being dealt with.

You could choose to keep or throw away the event messages and accounting records that you have used to create the file with. An example of discarding an accounting record after you have used it for what you needed is shown in this post.

If you keep the data you can come back to it later and create other files based on DISPLAY command output from MQEV, just as you can with DISPLAY command output from IBM MQ as discussed in this blog post.


Read more about MQEV, and download a copy, from the MQEV Web Page. If you don’t have a licence and would like to try out MQEV before deciding whether to buy then send an email to support@mqgem.com and a 1-month trial licence will be sent to you.

Advertisement

MQEV Facet Videos

We just uploaded a small selection of short videos about different facets of the MQGem product, MQEV.

We’ll be sharing one of these each Friday in the coming weeks, using the #FacetFriday tag (Twitter, Facebook, LinkedIn), but if you’d prefer to binge watch them on your own schedule, they can all be found in the MQEV Playlist of our YouTube Channel.

Here’s the first one you get you started:-


If you don’t have a licence and would like to try out MQEV then send an email to support@mqgem.com and a free, no-obligation, 1-month trial licence will be sent to you.

Managing Not Authorized (Type 4) Events

There are various types of IBM MQ Not Authorized events:

This post is focusing on the Type 4 events which report when a command, e.g. MQSC or PCF, is issued and fails because it is not authorized. When a user that does not have authority to display all the queues on a queue manager issues a command such as:-

DISPLAY QLOCAL(*) ALL

then you might end up with rather a lot of these event messages, one written for each queue that the user does not have display access to.

Unfortunately these event messages tell you very little. Their entire content is documented in IBM Docs as follows:-

Event data
QMgrName
Description: Name of the queue manager generating the event.
Identifier: MQCA_Q_MGR_NAME.
Data type: MQCFST.
Maximum length: MQ_Q_MGR_NAME_LENGTH.
Returned: Always.
ReasonQualifier
Description: Identifier for type 4 authority events.
Identifier: MQIACF_REASON_QUALIFIER.
Data type: MQCFIN.
Values:
MQRQ_CMD_NOT_AUTHORIZED
Command not authorized.
Returned: Always.
Command
Description: Command identifier. See the MQCFH header structure, described in Event message MQCFH (PCF header).
Identifier: MQIACF_COMMAND.
Data type: MQCFIN.
Returned: Always.
UserIdentifier
Description: User identifier that caused the authorization check.
Identifier: MQCACF_USER_IDENTIFIER.
Data type: MQCFST.
Maximum length: MQ_USER_ID_LENGTH.
Returned: Always.

One thing that is missing from these events is the object name the event is about. If you issue a generic display command you can get hundreds of identical events because there is nothing to tell you which event is for which object. Unfortunately there is also nothing to tell you that this was as a result of a generic display command either, just that it was a display command. You can infer that it was a generic display command because you get so many identical events from the same user id in a very short space of time.

It is really a shame that these actions even generate such events at all. The user asked to see all the objects they were allowed to see, and they were only given command responses for those objects they were allowed to see. Raising authority events for such an action doesn’t really seem warranted. Perhaps we should raise an IDEA with IBM to suggest this could be turned off.

Saving off these events to a different Event Stream

Another blog post covers how you can save off different MQ event messages to different streams in order to apply longer (or shorter) retention intervals to them, and these specific event messages do seem like an excellent candidate for that, this time though in order to have a much shorter retention interval.

As noted in that blog post, you might normally like to retain security related events for longer than the MQEV default of 90 days, but for these we might like a much shorter interval.

DEFINE EVSTREAM(SECURITY) TYPE(EVENTS) RETINTVL(120)
DEFINE EVSTREAM(DISCMDSEC) TYPE(EVENTS) RETINTVL(7)

Then the mqev.mqx script would be enhanced to include the following:-

*********************************************************************
* Function for processing an event                                  *
*********************************************************************
func MQEVEvent()
  if (event.EVTYPE = AUTHOR)
    if (event.EVREASON = AUTCMD AND +
        substr(event.SUMMARY,1,27) = "Command not Auth - Inquire ")
        _stream = "DISCMDSEC"
    else
      _stream = "SECURITY"
    endif
  endif
endfunc

While we could have checked the event.COMMAND attribute in the event against const.MQCMD_INQUIRE_Q and all the other MQCMD_* constants, that would have been a very long list and we’d have to maintain it too. Instead we use the event.SUMMARY string which will always begin with the same characters for an Inquire command and let the MQGem team keep MQEV up-to-date with any new commands that are added to IBM MQ.

Now that all these commands are being stored on a different stream we can view our other events without these in the way. We can also easily purge this stream and throw away these events whenever we want to, and MQEV will automatically throw them away for us after the retention interval on the stream expires.

Alternatively, if you just want to immediately throw away ALL of these DISPLAY events (whether from generic display commands or specific display commands) you could instead set the stream to null like this:-

_stream = "$null"

Event Storms

Another thing that MQEV does to assist you not being completely flooded by events such as this, is that it detects events that are identical. This is known as an event storm. These “Command Not Authorized” events fall squarely in this category since there isn’t even an object name in the event message.

By default, if MQEV sees more than 20 identical events in a 60 second time period, then it will begin to capture them as a repeated event. This means you will see 20 identical events, and then at the end of the 60 seconds, one more event with an EVREPEAT count showing how many more there were.

DISPLAY EVENTS(*) WHERE(EVREPEAT) ALL
EVQMGR(MQG1)        EVENTS(DISCMDSEC)   EVTIME(2022-08-18 17:38:44 (Local))      
EVREASON(AUTCMD)    EVTYPE(AUTHOR)      EVUSERID(mqgusr1)   EVOBJTYPE(CHANNEL)   
COMMAND(Inquire Channel)                EVENTID(00000068)   CFHCMD(44)           
CFHREASON(2035)     SUMMARY(Command not Auth - Inquire Channel - Channel:)       
RSNQUAL(4)          USERID(mqgusr1)     EVREPEAT(15)                             
_________________________________________________________________________________
EVQMGR(MQG1)        EVENTS(DISCMDSEC)   EVTIME(2022-08-18 17:38:37 (Local))      
EVREASON(AUTCMD)    EVTYPE(AUTHOR)      EVUSERID(mqgusr1)   EVOBJTYPE(QUEUE)     
COMMAND(Inquire Q)  EVENTID(00000067)   CFHCMD(44)          CFHREASON(2035)      
SUMMARY(Command not Auth - Inquire Q - Queue:)              RSNQUAL(4)           
USERID(mqgusr1)     EVREPEAT(113)                                                
_________________________________________________________________________________

You can alter these settings using an ALTER EV STORMTHR(number) STORMINT(seconds) command.


Drowning in MQ events? Let MQEV help.


Read more about MQEV, and download a copy, from the MQEV Web Page. If you don’t have a licence and would like to try out MQEV before deciding whether to buy then send an email to support@mqgem.com and a 1-month trial licence will be sent to you.

Graphing queue get/put rates

I want to graph the production and consumption rates of messages on a queue. What is the best way to do this?

There are two main ways of doing this, with MQGem Software tools, and these two methods have various pros and cons, which I would like to show and discuss in the post.

Graph Reset QStats

Here is a screen shot of a graph of the enqueue and dequeue values from a Reset QStats command. Instructions on how to create this graph will be given at the end of the blog post.

MO71 Graph produced from a monitor using the RESET QSTATS command

Graph Queue Statistics

Here is a screen shot of a graph of the total of successful puts and total of successful gets from Queue Statistics. Instructions on how to create this graph will be given at the end of the blog post.

MO71 Graph produced from MQEV Queue Statistics data

Which should I use?

There are different characteristics of these two ways of graphing what is in essence the same data. Let’s look at them side by side.

Reset QStats Queue Statistics
Polls the Command Server. Care should be taken if you are monitoring large numbers of queues at high frequencies using this method because of the incurred load on the queue manager. However, for single queue high frequency graphs, this may be more suitable as you would you not likely want to set Accounting and Statistics intervals to really short timeframes, such as a minute or less. Information is pushed by the Queue Manager as Accounting and Statistics messages, at configured intervals, and it therefore incurs less load on the queue manager.
Can only be graphed by one user as RESET QSTATS is destructive Can be graphed by any number of users
Using RESET QSTATS will interfere with the collection of Performance Events Using Queue Statistics does not interfere with any other data collection
Available on z/OS (turn on performance events to use it) and Distributed platforms Only available on Distributed platforms, not on z/OS
Configuration is very simple to monitor a small number of queues, but becomes unwieldly to monitor many/all queues Configuration requires a small amount of initial set up, but once set up, monitoring the entire queue manager is no more effort than monitoring a single queue
MO71 does not harden this data, so the data must processed/graphed immediately MQEV will keep the accounting data for as long as you tell it to – 3 months by default. At any time you can request to see historical data for any of your queues.
Can be done using only MO71 Requires MQEV

In summary, MO71 is an administration tool, that provides a few simple monitoring capabilities, but it is not a full blown monitoring tool. If you just want to see what happens to a few objects for a short period of time, say whilst diagnosing a specific problem, then it is a suitable solution. However, if you have more wide-spread, or longer-term monitoring requirements, particularly if there is a requirement to harden the data, then MQEV is the more appropriate choice, with MO71 providing the graphing interface on top of that data.

Creating Graph Using Reset QStats

In order to create a graph of integer values obtained by sending commands to the command server, you must first create a “monitor”. This is what sends the commands and collects the responses so that you have data to graph. Begin at the Monitor menu and choose Monitor List… From this dialog you can make a New… monitor from the right-click context menu and fill in the details of the command and attributes you wish to monitor. Here’s my monitor definition.

MO71 “monitor” definition

Once you have created it, start the monitor and it will begin to send command messages to the command server in order to collect the data.

Now you can graph the data which the monitor has collected. Begin at the Monitor menu and choose Graph… This will display the graphing canvas which is currently blank. Right click on the graphing canvas to set up your graph.

Any running “monitors” you have will be shown on the Monitors tab. Open up the twisty for your queue manager to see the sets of data you have available to graph. In my case I have chosen to collect data on two attributes from the Reset QStats command, so I see that I have a set of data for each attribute.

To add each set of data to the graph, click on the empty square to the left, to tick the line. At the same time you can also choose graph type, colour, thickness and so on. With graphs such as these, where both lines are quite similar, you may find it useful to invert the Msgs Out, which is what the graph illustrated did.

Having created your graph with appropriate sets of data and the colours you want, remember that there are other options on the “Graph” tab of this dialog. For example, you can give the graph an appropriate title, which will then be reflected in the graph window title.

Creating Graph Using Queue Statistics

To collect Queue Statistics for a single queue, you can enable it on that queue alone.

ALTER QLOCAL(WORK.Q1) STATQ(ON)

Alternatively, you can enable Queue Statistics for many/all queues by enabling it on the queue manager.

ALTER QMGR STATQ(ON)

Either way, the interval (in seconds) upon which statistics data will be pushed out by the queue manager is configured on the queue manager object.

ALTER QMGR STATINT(900)

MQEV will collect the statistics data for you. Ensure that MQEV is running and reading from the appropriate SYSTEM queue.

=mqev
RESUME EVQ(SYSTEM.ADMIN.STATISTICS.QUEUE)

Once you have MQEV collecting MQ Statistics data, you can graph the data that it has collection. Begin at the Monitor menu and choose Graph… This will display the graphing canvas which is currently blank. Right click on the graphing canvas to set up your graph.

For Queue Statistics data you should switch to the StatQ tab. Before you open up the twisty to see the data, configure the time period you wish to graph and the summation interval to be used. I have my interval set to 15 minutes to match the collection interval. My time period was chosen to make the graph look pretty! You should do this first because these settings are used by MO71 when it sends commands to MQEV to get the data to graph.

Now open up the twisty for your queue manager to see the sets of data you have available to graph. You will see all the integer values available from Queue Statistics records (this screenshot is truncated for reasons for brevity).

To add each set of data to the graph, click on the empty square to the left, to tick the line. At the same time you can also choose graph type, colour, thickness and so on. With graphs such as these, where both lines are quite similar, you may find it useful to invert the gets, which is what the graph illustrated did.

Having created your graph with appropriate sets of data and the colours you want, remember that there are other options on the “Graph” tab of this dialog. For example, you can give the graph an appropriate title, which will then be reflected in the graph window title.

 
Watch this video of MO71 graphing MQEV data.


If you don’t have a licence and would like to try out MO71 and MQEV then send an email to support@mqgem.com and a 1-month trial licence will be sent to you.

Latest MQEV and MQSCX released on z/OS

Our recently released versions of MQEV V9.2.2 and MQSCX V9.2.1 are now also available on z/OS.

Read more about these versions here:-


The new versions can be downloaded from the MQEV Download Page and the MQSCX Download Page. Any current licensed users of these products can run the new version on their existing licence. If you don’t have a licence and would like to try out MQEV or MQSCX then send an email to support@mqgem.com and a 1-month trial licence will be sent to you.

Post Event Data to HTTP Service

In an earlier blog post we looked at how MQEV emitters can write out your IBM MQ Event, Accounting and Statistics data, to files or queues for post-processing. One of the examples of the types of post-processing you might use this for was to upload JSON format data to an HTTP centralised service such as Elastic. This post is going to look at that in more detail.

As a user of MQ, you will understand the benefits of decoupling applications by using a queue. The example of posting event data to an HTTP service is a good example of a case where this decoupling is very useful. Rather than have MQEV do the HTTP post as part of consuming the event message, it instead writes the JSON data to a queue. This means that the mainline processing of event messages is not impacted if the HTTP post cannot be completed right at this moment. Then we have another application which reads the messages from that queue and posts them to the HTTP service.

This application can be triggered so that it starts up when it needs to. If the data is from event messages, these arrive on a non-deterministic pattern. If it’s Statistics data, the data arrives every 30 minutes, say. Accounting data is a combination of the two, the data will arrive at set intervals, but also when the application disconnects. In all cases, there is likely to be a lot of waiting around, so triggering is perfect.

MQGem supplies a GetPost Python application that reads from the queue, checks that the data is valid JSON, and then posts the data to the specified URL. This application meets the needs for this particular MQEV use case, but might be useful for other scenarios too. We offer it as a free application to anyone who can find a use for it.

We chose Python for this application because we needed a language that had MQ libraries, HTTP libraries and JSON libraries. Python met the requirements, and allows a single application to be written that can run on many of the main business platforms[1].

The GetPost application currently provides two ways to post the data to a URL.

  1. General Purpose POST of JSON data
    MQEV emits data in JSON. This data can be posted exactly as it appears in the message data, but does require the use of the MQEV emitter attribute GROUPING(SINGLE). This is a basic form of Get->Post and will be suitable for a number of HTTP services. It works with Elastic too.
  2. Elastic Bulk Data POST
    This mode is specifically for use with Elastic and utilises the Elastic Bulk API which can POST multiple items with a single HTTP POST. This is especially useful when you have emitters using GROUPING(ORIGINAL) on Accounting and Statistics data such that there are multiple items in a single emitted message. This provides an efficiency not available with the general purpose method which has to post each message as a separate HTTP POST.

    This mode also takes the unique identifier from the MQEV message data and uses that as the unique Elastic _id allowing the detection of duplicate POSTS. This is very useful since, of course, an HTTP POST is not able to be transactional. It is therefore preferential to use this mode even when no grouping is in use.

Once the data is input into Elastic, you can graph it and display in many different ways. However, likely as not, if you are interested in Elastic, you know more about this than we do!

Graph of Queue Statistics showing the maxdepth attribute for each queue over a number of intervals


The new version of MQEV can be downloaded from the MQEV Download Page, and the User Manual for MQEV also describes the GetPost application. The GetPost application can be downloaded from the GetPost Webpage. Any current licensed users of MQEV can run the new version on their existing licence. If you don’t have a licence and would like to try out MQEV then send an email to support@mqgem.com and a 1-month trial licence will be sent to you.


Footnotes:

  1. Only today, I discovered that Python is also available on z/OS when I read Top 8 Reasons for using Python instead of REXX for z/OS, so I will have to look into that too. And more recently still, we now have PyMQI libraries for z/OS thanks to Colin Paice.

MQEV Emitters

A new feature added to MQEV V9.2.2 is emitters. This feature allows you to configure MQEV to emit the data it has gathered about events, accounting and statistics, to files or queues in a chosen format, JSON, CSV or MQSC. This can make it easy to ingest this data into downstream processes.

You might want to do this because:-

  • You have a centralised system for capturing such data, such as Elastic or Splunk.
  • You want to post-process and graph numeric data such as Accounting and Statistics.
  • You wish to maintain a long term archive of event data.
  • You can also use it to debug your queue manager event settings and MQEV config to easily see the events being captured and processed as they come in, for example by directing the incoming events to a queue in MQSC format, and having an application sitting in a get wait on that queue.

This post is going to describe how to set them up.

EV Streams

The first thing to look at is EV Streams. This is the control point to break out the different types of data you are collecting. By default, MQEV gives you one for each type of data as shown in the table below, but you can create more detailed breakdowns than that, as described in an earlier post Retaining Events with MQEV. In our diagram we have a stream called CFGEV where we have stored all the configuration events.

Type of data Default EV Stream Name Grouping
Events $EVENTS Always single
Accounting MQI $ACCTMQI Always single
Accounting Queue $ACCTQ Arrays possible
Statistics MQI $STATMQI Always single
Statistics Queue $STATQ Arrays possible
Statistics Channel $STATCHL Arrays possible

EV Streams are also the control point where you configure EV Emitters. You’ll want a particular type of data to be emitted in a particular way, or not at all.

EV Emitters

An EV Emitter is a new configuration object in MQEV that details the file or queue you want data to be emitted to, the format (JSON, CSV, or MQSC) and various other controls on that data.Here’s an example command to create an emitter to a queue.

DEFINE EVEMIT(JSON.TO.Q) QUEUE(EVENTS.JSON.EMIT) +
       DESCR('Emit Event data in JSON to a queue') +
       FORMAT(JSON) +
       READER(PROGRAM) +
       GROUPING(SINGLE) +
       MSGPERS(YES) +
       ONERROR(SUSPEND)

Let’s look in more detail at this configuration.

  • We’ve given it a name that explains it’s purpose, it’s to be used to emit JSON format data to a queue.
  • This emitter will write the data as messages to a queue, and we’ve named the queue here to describe the contents.
  • The emitter knows to write JSON messages because of the FORMAT attribute.
  • The READER attribute will change the style the output is written in. Either quite compressed, with very few new lines, and thus hard for a human to read, or spaced out, indented, plenty of newlines, to make it easy for a human to read. The content is the same for either READER(PROGRAM) or READER(HUMAN), it’s just the whitespace that changes, and thus can make the message a bit bigger.
  • For some types of data (as shown in the table above), arrays of records are possible, and can be a more efficient way to work with the data. If you want to force each record to be an individual emitted message, you can use GROUPING(SINGLE). For the types where the data is always single anyway, this attribute has no effect.
  • Emitted messages can take their persistence from the original event message that was created by the queue manager – MSGPERS(ASMSG) – or alternatively you can set it explicitly as this example shows.
  • Finally, you need to decide what behaviour you want in case of an error, say your emitter queue fills up. You can choose to suspend reading from the event queue while you sort out the issue, or you can choose to discard the emitted message and continue processing from the event queue.

Having created your emitter definition, you then associate it with the stream as follows:-

ALTER EVSTREAM(CFGEV) EVEMIT(JSON.TO.Q)

Emitting to files

In addition to what we have seen about emitters, there is a little more to say when emitting data to a file. When emitting to a queue, each item is written as a message to the queue. When emitting to a file, you probably want multiple items to end up in the same file. For example, you might want a CSV file containing all the Queue Statistics Data for one day, giving you several rows in the one file, and making it easy to graph that data.

One of the things that dictates the way the file is used is the name of the file, more specifically the inserts in the file name.

Our above example of capturing a days worth of data in a single file, could be done as follows using the %C insert which is for the current date:-

DEFINE EVEMIT(CSV.TO.FL) FILE('C:\MQGem\StatQ%C.csv') +
       DESCR('Emit StatQ data in CSV to a file') +
       FORMAT(CSV) +
       READER(PROGRAM)

It is possible to force each item to create a new file, by using the %i insert, which is just an index on the file name.

There are a wide selection of inserts which you can see full details about in the MQEV User Manual.

Downstream processing

Having got MQEV to write data to a file, you can then process it using tools that understand the particular file type, for example a spreadsheet to use a CSV file.

Having got MQEV to write data to a queue, you need to have something processing that queue. If your data is in JSON format and you need to upload it to a central service that listens on an HTTP listener, you may find the GetPost application useful. Read more about this application here.

Summary

So we have expanded a little on the detail of how to configure an MQEV Emitter. Our flow picture has expanded to show more of the steps.


The new version can be downloaded from the MQEV Download Page. Any current licensed users of MQEV can run the new version on their existing licence. If you don’t have a licence and would like to try out MQEV then send an email to support@mqgem.com and a 1-month trial licence will be sent to you.

MQEV version 9.2.2 is released

MQGem Software is pleased to announce that a new version of MQEV, our Event Processing tool for IBM MQ, is now available.

The main features of the release are as follows:-

Addition of Emitters

You can use MQEV to emit events, accounting and statistics data, ready to push to a centralised store such as Elastic. Alternatively, you can have all your data written out as files ready to import into a spreadsheet. Both of these cases, and others you may have, are catered for by MQEV emitters. These can be configured in MQEV to write out data from events, accounting or statistics, to queues or files, in JSON, CSV or MQSC format.

You can choose any subset of your events, by first setting up your data to different streams, MQEV’s control point for your data. Then the set of data that goes to one stream, is emitted based on the configuration associated with that stream.

Read more about MQEV Emitters in this blog post.

Improvements for large sets of data

Commands to interrogate MQEV already have limits on them so that if you simply type in DISPLAY EVENTS(*) you don’t end up with an unwieldy number of results. By default results are limited to only return 100 results (equivalent of typing MAXRESP(100)) and also to only return those results from the past day (equivalent of typing FROM(-24h)). However, if you have millions of records captured in a day, more likely for Accounting and Statistics than Events, and you request a totalled answer, by using a command something like the following, then that might be a very large set of data that needed to be processed and totalled to give you your answer as a single response.

DISPLAY ACCTMQI(*) SUM(TOTAL)

To protect against inadvertent processing of large sets of data, MQEV now has a second limiting concept for DISPLAY commands, called MaxRecords (MAXRECS), which controls the number of records that will be processed in a DISPLAY command. If there were more records than that, the command would warn you that the record set had been limited, and you could consider whether you really wanted to total up more than a million records, and, if so, raise the limit, or alternatively, filter the records used so that the set of records was less than a million, perhaps by only looking at the last 8 hours, instead of the last 24 hours, for example.

For many users, the need to increase MAXRECS from it’s default of 1 million will be unnecessary, but for those users with these large sets of data, they can increase the default used for MAXRECS in the ALTER EV command, using the DEFMAXRECS attribute.

Support for hex types in MQEV WHERE clause

A number of fields in events, accounting and statistics records are hexadecimal, such as ConnectionIds, or Channel error codes. It is now possible to filter on these types in the MQEV WHERE clause.

DISPLAY ACCTMQI(*) SUM(NONE) ALL
        WHERE(CONNID EQ 0x'414D51434D514732202020202020202010CAE36001CCCD23')

See build date when running MQEV in background

When you run MQEV in the foreground, the first thing it does is report the build date.

MQEV Version:9.2.2 (64 Bit) Build Date:Dec 10 2021

However, when running it in the background, or as a service, you can’t just look in that foreground window. It is written in the log file, but that’s not your normal way of interacting with MQEV. So now you can discover this using the DISPLAY EV command.

NAME(MQEV)          DESCR(MQGem Software IBM MQ Event Processor)
STARTTI(2021-12-10 22:16:50 (Local))  BUILD(Dec 10 2021)
CMDLEVEL(922)

Command Level support up to 924

MQEV now supports queue managers up to the most recently released command level of 924.


The new version can be downloaded from the MQEV Download Page. Any current licensed users of MQEV can run the new version on their existing licence. If you don’t have a licence and would like to try out MQEV then send an email to support@mqgem.com and a 1-month trial licence will be sent to you.

Are you using any back-level clients?

We have a list of questions that MQEV can help you to answer. One of them is:-

  • What version of MQ client are connecting to my Queue Manager? Are any of them old and in need of an update?

In this post I want to show you how to answer that question.

You may be aware that the version of the MQ client is something that is flowed across to the server. Originally it was just used to write out in trace for IBM Service; then it was added as output on DISPLAY CHSTATUS; and then in MQ V8.0.0.3 it was given to channel exits, and then in MQ V9.1.0, it was added to MQI accounting records.

Capturing this data using accounting records is an excellent way to do it because you don’t have to worry about missing connections from applications that are so short lived that you can’t catch them with a DISPLAY CHSTATUS command. An MQI accounting record is written for each application at disconnect time (explicit MQDISC or implicit), but also periodically every ACCTINT if the connection duration is long enough and the application is doing any MQI work.

To do this, you turn on MQI Accounting records, using the following MQSC command.

ALTER QMGR ACCTMQI(ON)

Then tell MQEV to monitor the accounting queue with the following MQEV command.

RESUME EVQ(SYSTEM.ADMIN.ACCOUNTING.QUEUE)

Then you will get data for each application that connects to your queue manager. At the start of each record of this type will be various identifying fields about the application. For example the application name, the channel name it connected across (assuming it is a client), and the connection name it came from. In addition you will see the remote product and remote version of the client.

MO71 display Accounting MQI records for clients with different Remote Versions

If you don’t have any reason to collect MQI accounting, apart from answering this question, you could simply raise an Alert when you see a client older than a certain level, and then discard the rest of the data. Here’s how you would do that in the mqev.mqx script.

*********************************************************************
* Function for processing an Accounting MQI message                 *
*********************************************************************
func MQEVAcctMQI()
  if (data.CHANNEL)
    if (data.RVERSION < "09020000")
      @AlertStr  = "Application: " + data.ACCTMQI + -
                 " from CONNAME: " + data.CONNAME + -
                 " has a back-level client version: " + data.RVERSION
      @AlertObj  = data.CHANNEL
      @AlertQMgr = data.EVQMGR
      ADD EVALERT TEXT('<@AlertStr>') CATEGORY(CLIENTVER) +
          EVOBJNAME('<@AlertObj>') EVOBJTYPE(CHANNEL)     +
          SEVERITY(WARN) EVQMGR('<@AlertQMgr>') REPLACE
    endif
  endif
  *******************************************************************
  * Discard the MQI data - I have got all I needed from it          *
  *******************************************************************
  _stream = "$null"
endfunc

MQEV alerts can be displayed using a GUI such as MO71 or a command line tool such as MQSCX. When using MO71, if the alert list dialog is open, it will be automatically refreshed every time a new alert is added since MO71 subscribes to a topic that gets a publication with every new alert. The use of REPLACE in the above script means that only one alert will be written for each unique client. So if the same client application connects 100s of times, there will still only be one alert. Once the administrator believes the back-level client has been upgraded, the alert can be deleted from MQEV, and should it appear again, you know the problem is not yet fixed.

MO71 list of MQEV Alerts showing some back-level clients

You can read more about MQEV Alerts in this blog post.


Read more about MQEV, and download a copy, from the MQEV Web Page. If you don’t have a licence and would like to try out MQEV before deciding whether to buy then send an email to support@mqgem.com and a 1-month trial licence will be sent to you.

Running MQEV as an MQ SERVICE

If you have IBM MQ Event Switches enabled, whenever your queue manager is running, you should have a program consuming the event messages that are generated. If you don’t, your event queues will fill up, and newer event messages will be silently discarded by the queue manager.

One way to ensure that there is always a program running, and consuming these messages, while the queue manager is running, is to define that program as an MQ SERVICE object, and let the queue manager start it up. You can do this with any program, and MQEV is no exception.

In MQEV V9.2.0, we added an additional flag, -k, that tells MQEV it is being run in this mode which changes the way it behaves a little:-

  • It will run in background mode (just as if you had specified -b)
  • It will not retry connection failures, but instead just end. It will be running as a locally bound connection, and so these failures will be as a result of the queue manager ending (normally or abnormally). When the queue manager next starts up, so will MQEV. It was possible to mimic this prior to MQEV V9.2.0 by using -r none, but that had the effect of suppressing retry of all failures. -k only suppresses connection failure retry.

So to run MQEV as an MQ service, you need a definition like this on your queue manager (along with the two queues that MQEV uses).

DEFINE SERVICE(MQGEM.MQEV) +
       DESCR('MQGem Software Event Monitor') +
       CONTROL(QMGR) SERVTYPE(SERVER) +
       STARTCMD('mqev') STARTARG('-m +QMNAME+ -k') +
     [ STOPCMD('mqscx') STOPARG('-m +QMNAME+ -f -C "=mqev;STOP EV"') ]

To explain the various parameters:-

  • CONTROL(QMGR) ensures the program is started when the queue manager starts up.
  • SERVTYPE(SERVER) says that there is only one instance of this SERVICE running at any one time, and allows you to use the DISPLAY SVSTATUS command to check whether it is running.

    MO71 showing a list of Service Status containing MQEV

  • The STARTCMD names the program, which in my case is in the path, and STARTARG provides the parameters. The use of +QMNAME+ is a short-hand on SERVICE objects for the name of the owning queue manager. This means you don’t have to change the definition to copy it to other queue managers.
  • The STOPCMD is optional, because MQEV will end when it detects the queue manager is quiescing, or gets any other connection failure.
    • It might be useful to code the STOPCMD and STOPARG if you are developing your event scripts, for easy stop and restart of MQEV using the STOP SERVICE and START SERVICE MQSC commands. However, you can just as easily issue the STOP EV command via MQSCX or MO71 interactively.
    • The STOPARG are parameters to the MQSCX program, and tell it to run the initial command (-C) of =mqev followed by STOP EV.

Read more about MQEV, and download a copy, from the MQEV Web Page. If you don’t have a licence and would like to try out MQEV before deciding whether to buy then send an email to support@mqgem.com and a 1-month trial licence will be sent to you.