Application Activity Trace Viewer

IBM MQ has a feature called Application Activity Trace, which allows you to trace what an application is doing, see the progression of MQ API calls, the objects utilised, the options used, the time taken for each call and so on.

To get the most out of this feature, which writes its output as PCF messages, you need to be able to, not only read the content of those PCF messages, but display them in a way that allows you to drill down into the data, exclude data from your view to allow you to focus on what’s important, search for particular objects in use, or particular reason codes, and so on.

To this end, MO71 now has an Application Activity Trace viewer. At it’s simplest, it allows you to see the progression of MQ API calls, just like the IBM-supplied sample amqsact. However it can do so much more than that.

Activity Trace Output

MO71 Activity Trace Output

Drilling down

Rather than having to decide to view every MQ API call in high detail, the viewer allows you to drill down into the verb you are interested in, without the confusion of seeing high detail of everything else at the same time.

Activity Trace Drill down MQINQ

MO71 Activity Trace – Drill down into MQINQ

Activity Trace API Selection

MO71 Activity Trace API Selection

Filter the output

With a trace that spans many pages full, even when only showing one line per verb, you may wish to reduce the clutter on your screen further. You can filter the output in many different ways to see what you’re looking for. For example, you could choose to hide all the MQPUTs and MQGETs and focus on the other calls, like the MQOPENs.

Activity Trace No PUTs and GETs

MO71 Activity Trace with PUTs and GETs excluded

You can filter the output to focus on a particular object or objects that you are interested in, or a particular process id. In fact there are many different factors that you can used to filter down the data to make is easier to view.

Activity Trace Settings

MO71 Activity Trace Settings Tab with filtering

Health-checking your application

There are a number of behaviours that MO71 can check for in your application. These are listed on the Health tab, and if any issues are found you will see them highlighted with a red exclamation mark. You can display the instances in a separate window which then allows you to jump into the main output window at the point where the issue was detected.

Activity Trace Health

MO71 Activity Trace Health Tab

The current list of health issues is not final, and if you have any other issues you would like to see MO71 check for, please comment below or get in touch in the usual ways.

The Application Activity Trace viewer in MO71 will help you make sense of, and get great insight from, your tracing of your applications.


The new version can be downloaded from the MO71 Download Page. Any current licensed users of MO71 can run the new version on their existing licence. If you don’t have a licence and would like to try out MO71 then send an email to support@mqgem.com and a 1-month trial licence will be sent to you.

Migrating a Queue Manager?

If you’re planning to migrate a queue manager, from one machine to another, or perhaps you’re consolidating some queue managers onto an MQ Appliance, you are probably aware of, and have even become familiar with, the steps required to export your object definitions and recreate them on the new queue manager. There are a number of tools available that can do this for you – including those from MQGem.

What about your messages though?

While it’s generally a good idea to reduce the load, and drain off as many messages as you can prior to migrating a queue manager. For example if it’s in a cluster, suspend it from the cluster before the move, to reduce the number of messages heading its way. Let all the applications drain the queues of messages and so on. However, it’s not always possible to drain every last message from all your queues.

QLOAD Offload all queuesIn the latest version of QLOAD, V9.0.1, there’s a new feature that will help out when migrating a queue manager. It allows you to unload the messages from all your queues, with one command. And then you can load the messages onto the queues on another queue manager, with one command (or piecemeal if you prefer). You would combine this with your favoured object definition export tool, and before loading the messages, you’d first recreate the queues on the new queue manager with the exported commands.

Here’s an example of QLOAD unloading all the queues on my queue manager.

qload -m MQG1 -i* -f*

The command will produce the following output to show what has been unloaded. Using the -i (lower case) flag means that the messages are only browsed on the queues and are not destructively removed.

APP1.INPUT                                           2  Done.
APP1.RESULT                                         10  Done.
APP2.INPUT                                           1  Done.
APP2.RESULT                                          4  Done.
Q1                                                  28  Done.
Q2                                                  42  Done.
SYSTEM.ADMIN.QMGR.EVENT                              7  Done.
SYSTEM.AUTH.DATA.QUEUE                             126  Done.
SYSTEM.CHANNEL.SYNCQ                                 3  Done.
SYSTEM.CHLAUTH.DATA.QUEUE                            5  Done.
SYSTEM.CLUSTER.REPOSITORY.QUEUE                      3  Done.
SYSTEM.DURABLE.SUBSCRIBER.QUEUE                      1  Done.
SYSTEM.HIERARCHY.STATE                               2  Done.
SYSTEM.INTER.QMGR.FANREQ                             1  Done.
SYSTEM.RETAINED.PUB.QUEUE                            2  Done.
WORK.REQUEST                                         7  Done.

Total : 7 Queues, 94 Messages
 plus : 9 System Queues, 150 Messages

Listing the directory where I ran the qload command, I can now see I have a file for each queue, with an extension .qld. If you prefer to have a different extension then you can alter the command accordingly. For example, use -f*.txt.

I can copy these files to another machine, or simply use a client connection to the other machine accordingly. Then I can run the following QLOAD command to load the messages onto the new queue manager.

qload -m MQG1 -o* -f*

QLOAD will make some checks when you run a load for multiple queues in this way. It will check that all the queues exist that it has files for in the directory matching the file pattern you specified (which assumes an extension of .qld if you just use ‘*’), and it will check that all those queues are empty. If it finds any problems it will report as follows:-

APP1.RESULT                            RC(2085) Unknown object name.
APP2.RESULT                            Not empty.

There are potential problems with these queues.
Are you sure you want to continue?

This lets you know to go and define the missing queues if you need them – there are messages to go onto them so the assumption is that you do need them. It also warns you of non-empty queues. Now if you’ve already started using this queue manager, you might be expecting this situation, but otherwise, you should rectify it, and then re-run QLOAD.

APP1.INPUT                                       2     Done.
APP1.RESULT                                      10    Done.
APP2.INPUT                                       1     Done.
APP2.RESULT                                      4     Done.
Q1                                               28    Done.
Q2                                               42    Done.
WORK.REQUEST                                     7     Done.

Total : 7 Queues, 94 Messages

You’ll notice that this output does not mention the SYSTEM queues that were offloaded. A generic upload will not upload most SYSTEM queues.
SYSTEM.CLUSTER.TRANSMIT.QUEUE and SYSTEM.DEAD.LETTER.QUEUE are the exceptions. The messages were unloaded to files though, so if you really need to load them you can do so the traditional way, by specifying the full queue name on the command and not using the generic upload.


The new version can be downloaded from the QLOAD Download Page. Any current licensed users of QLOAD can run the new version on their existing licence. If you don’t have a licence and would like to try out QLOAD then send an email to support@mqgem.com and a 1-month trial licence will be sent to you.

Running the Trigger Monitor as a SERVICE

There was a recent update to MO71 that allowed multiple SERVICE objects to be edited at once.

The example used in the screenshot was of the trigger monitor being run as a service, and is straight out of Knowledge Center (with the exception of a more meaningful object name).

It uses the provided amqsstop program as recommended too. The parameters that amqsstop expects are provided in the STOPARG which include the +MQ_SERVER_PID+ which is a token representing the process id of the process started by the STARTCMD and STARTARG arguments.

I was playing around with this SERVICE object a little more today and discovered that the STOP SERVICE command doesn’t work. This post covers what I discovered and how to fix it.

You’ll note from the screen shot that I’m running a 64-bit Windows queue manager – you can tell that from the path of the amqsstop program which is in the bin64 directory. However, I used the runmqtrm program from the bin directory. This is no doubt a migratory aid for those users that had scripts etc starting the trigger monitor from that location prior to the Windows queue manager becoming a 64-bit entity.

Having started my trigger monitor with the above definition, I can see it’s status using the DISPLAY SVSTATUS command.

AMQ8632: Display service status details.
   SERVICE(TRIGGER.MONITOR)                STATUS(RUNNING)
   PID(3384)                               SERVTYPE(SERVER)
   STARTDA(2017-06-17)                     STARTTI(11.40.55)
   CONTROL(QMGR)                           STARTCMD(C:\mqm8004\bin/runmqtrm)
   STARTARG(-m MQG1 -q ACCOUNTS.INITQ)     STOPCMD(C:\mqm8004\bin64/amqsstop)
   STOPARG(-m MQG1 -p 3384)             
   DESCR(Trigger Monitor Service Auto Started with QMgr)
   STDOUT( )                               STDERR( )

Part of this display is the process ID of the trigger monitor, and you can also see that the replaceable insert +MQ_PROCESS_ID+ in the STOPARG attribute has been replaced with the same PID.

When you issue the MQ command STOP SERVICE(TRIGGER.MONITOR) it issues a PCF Inquire Connections command with a WHERE clause asking for all those connections where the PID is 3384. You can see in the MQ trace that the answer which comes back is MQRCCF_NONE_FOUND.

Now I know the trigger monitor is running so I find it myself in a DISPLAY CONN command and I see this:-

AMQ8276: Display Connection details.
   CONN(876C445920002201)                
   EXTCONN(414D51434D5147312020202020202020)
   TYPE(*)                               
   PID(4604)                               TID(1) 
   APPLDESC(WebSphere MQ Trigger Monitor)
   APPLTAG(:\mqm8004\bin64\runmqtrm.exe)   APPLTYPE(SYSTEM)
   ASTATE(NONE)                            CHANNEL( )
   CLIENTID( )                             CONNAME( )
   CONNOPTS(MQCNO_SHARED_BINDING)          USERID(MUSR_MQADMIN)
   UOWLOG( )                               UOWSTDA(2017-06-17)
   UOWSTTI(11.40.55)                       UOWLOGDA( )
   UOWLOGTI( )                             URTYPE(QMGR)
   EXTURID(XA_FORMATID[] XA_GTRID[] XA_BQUAL[])
   QMURID(0.20482)                         UOWSTATE(ACTIVE)

So there are two interesting things in this output. Firstly the PID is different. Secondly, it’s the bin64 version of runmqtrm. There’s no sign of the bin version of runmqtrm with PID(3384) anywhere in DISPLAY CONN. So I guess it didn’t make a connection to the queue manager.

Next thing to check out is the processes that the Windows OS thinks are running. I look for and find both PID(3384) and PID(4604).

runmqtrm processes

Two processes running called runmqtrm

So it seems that the runmqtrm in the bin directory is not a copy of the one in the bin64 directory, but something else that starts the bin64 version of runmqtrm. This means that amqsstop doesn’t work because it is trying to find the first process which never connected to the queue manager.

The fix to get your Trigger Monitor Service definition to work again with a STOP SERVICE command is to use the bin64 version of runmqtrm directly in the STARTCMD and avoid this double hop which leaves you with two processes running unnecessarily.

DEFINE SERVICE(TRIGGER.MONITOR) +
       SERVTYPE(SERVER) CONTROL(QMGR) +
       DESCR('Trigger Monitor Service Auto Started with QMgr') +
       STARTCMD('+MQ_INSTALL_PATH+bin64\runmqtrm') +
       STARTARG('-m +QMNAME+ -q ACCOUNTS.INITQ') +
       STOPCMD('+MQ_INSTALL_PATH+bin64\amqsstop') +
       STOPARG('-m +QMNAME+ -p +MQ_SERVER_PID+')

You don’t have the same problem on Unixes, because there aren’t the two bin directories on those platforms. So this is very specific to Windows.

Really it’s a shame that there isn’t a replaceable insert something like +MQ_BIN_DIR_PATH+ so that these platform differences would be completely removed from the SERVICE object definition. But I suppose you could make one yourself and put it into the service.env file.


IBM Certified SpecialistIBM Champion 2017 Cloud

Morag Hughson
IBM Champion 2017 – Cloud
IBM Certified System Administrator – MQ V8.0
Find her on: LinkedIn: http://uk.linkedin.com/in/moraghughson Twitter: https://twitter.com/MoragHughson SlideShare: http://www.slideshare.net/moraghughson developerWorks: https://www.ibm.com/developerworks/community/profiles/html/profileView.do?userid=110000EQPN

IBM MQ and MQ Appliance News – May 2017

On Tuesday May 30th, IBM Hursley made available the next in the series of Continuous Delivery releases for IBM MQ V9.0 and the MQ Appliance. IBM MQ V9.0.3 is now available.

Downloading IBM MQ Version 9.0.3 Continuous Delivery

This was announced on z/OS VUE:-

Links of interest:-


We’ll collect up any other links about the new release as we find them and put them all here.

IBM MQ V9 LTS FixPack 1

IBM recently shipped the first Fix Pack for the V9.0.0 Long Term Support (LTS) release.

Downloading IBM MQ Version 9.0.0.1

Spotted by one of our eagle-eyed followers, this document indicates:-

IBM MQ Version 9.0.0, Fix Pack 1 is released only on AIX, IBMi, Linux, and Windows. It is not released on HP-UX or Solaris.

EDIT: Fix Pack 1 is now available on HP-UX on Itanium, Solaris on SPARC, and Solaris on x86 64 as of 15 June 2017. Download from the above link.

We asked IBM why this was the case, and here is the answer.

9.0.0.1 was not shipped on HP-UX and Solaris due to an ongoing quality issue in the JVM on those platforms. We expect 9.0.0 LTS maintenance to be available on these platforms in the future. For more info [on the JVM quality issue on those platforms], head here: Oracle Bug Report: JDK-8175251 : Failed to load RSA private key from pkcs12.



Mini-Health Check for your Queue Manager

MO71 HealthcheckYou can pay expert IBM MQ consultants to do a full health check of your queue managers, but in between such events, or even in preparation for such events, how about running a mini-health check of your queue managers without the expense of consultation hours?

MO71 will look for a number of problems in your set of queue managers. Not just problems in an individual queue manager, but also the resolution of the various IBM MQ objects that refer to other objects, for example, from QREMOTE to transmission queue to sender channel to receiver channel to target queue across multiple queue managers.

MO71 HealthCheck Queue Resolution

MO71 will follow the resolution of your queues across queue managers.

Do you have a naming convention? Great! Do all your queue managers stick to it? MO71 will check that for you too.

MO71 HealthCheck Queue Naming

Tell MO71 your naming convention and it will check whether you have any objects not following it.

In order to have MO71 run these checks against your queue managers, open the Network view, and after selecting the list of queue managers you are interested in from the Queue Managers list, look at the Verify view. For each queue manager, you can open up the twisty labeled ‘ Problems’ to see the various things MO71 has detected.

MO71 HealthCheck Network View

Chosen queue managers in the left window; Verify view with problems in the right window.

If there are any of the problems that you don’t want to include in the check, you can deselect them from the Problems list. This is also the view that allows you to see all the problems that are looked for by MO71.

MO71 HealthCheck Problem Selection

The current list of problems checked for by MO71. Can you think of any others you would like to see?

If you have fixed any of the problems, you can refresh the data that the analysis is based on from the Action menu to see if you’ve fixed them all. Alternatively you can click on the database icon next to the queue manager name to only refresh the data from a single queue manager. If you’ve changed the problems being checked for, you can also Re-analyse the problems without retrieving new data from the queue manager.

MO71 HealthCheck Refresh Data

Use the Action menu to request data is Re-analysed, or refreshed from the queue manager.

As you can see, there are quite a number of things that MO71 can check for you, and it’s quick and easy. You simply need to tell MO71 where your queue manager is by adding it as a location into the tool, and then you can use it to detect these problems.


This is not a new feature of MO71. You already have it in the version you’re using. However, we do add new problems to be checked for from time to time. If you would like to see other things checked by MO71 to add to a mini-health check, please drop us a line or add a comment below.


If you don’t have a licence and would like to try out MO71 then send an email to support@mqgem.com and a 1-month trial licence will be sent to you.

What’s in Command Levels 90x

MQ90x StairsIBM MQ released Long Term Support release V9.0.0 back in June 2016 which had a Command Level of 900. The subsequent Continuous Delivery releases, V9.0.1, V9.0.2 and V9.0.3 have each introduced their own Command Levels, 901, 902 and 903 respectively.

This post captures the changes that are available in each of those Command Levels.

Release Command Level Features protected by Command Level – details below
V9.0.0.0 900 AMS Protection Policy enhancement – Confidentiality Policy
LDAP Authorization on Windows
V9.0.1 901 No changes protected by Command Level
V9.0.2 902 Log management features
V9.0.3 903 No changes protected by Command Level

AMS Protection Policy enhancement – Confidentiality Policy

With the introduction of Confidentiality Policies in Command Level 900, there is a new attribute on the Set Policy command. A confidentiality policy has no signature algorithm, but does have a encryption algorithm. The Key Reuse feature is applicable to this type of policy. Jon Rumsey has a great write-up of this IBM MQ V9 feature on the MQDev blog, MQ V9 Fast encrypted messages with MQ – Introducing AMS Confidentiality Policies.

AMS Policy

New Attribute MQSC name
See SET POLICY
Look for KC 9000 indicator
PCF constant and values
See Set Policy
Look for KC 9000 indicator
Key Reuse

KEYREUSE

  • DISABLED
  • UNLIMITED
  • 1 – 9999999

MQIA_KEY_REUSE_COUNT (267)

  • MQKEY_REUSE_DISABLED (0)
  • MQKEY_REUSE_UNLIMITED (-1)
  • 1 – 9999999

LDAP Authorization on Windows

Introduced in Command Level 801 on Unix, this feature extended the V8.0.0 Connection Authentication feature which checked your user ID and password, to allow LDAP authorization as well. The fields now available on Windows are the same as those noted in the earlier post for Command Level 801, and are not repeated here.

Log management

With the introduction of Automatic management of linear log extents, and Automatic writing of media images, in Command Level 902, there are new attributes on the queue manager object, queue manager status, and one on queue objects. Mark Whitlock has written about this in an MQDev Blog Post: Logger enhancements for MQ v9.0.2.

Queue Manager Object

New Attribute MQSC name
See ALTER QMGR
Look for KC 902 indicator
PCF constant and values
See Change Queue Manager
Look for KC 902 indicator
Image Schedule

IMGSCHED

  • AUTO
  • MANUAL

MQIA_MEDIA_IMAGE_SCHEDULING (268)

  • MQMEDIMGSCHED_AUTO (1)
  • MQMEDIMGSCHED_MANUAL (0)
Image Interval

IMGINTVL

  • 1 – 999 999 999
  • OFF

MQIA_MEDIA_IMAGE_INTERVAL (269)

  • 1 – 999 999 999
  • MQMEDIMGINTVL_OFF (0)
Image Log Length

IMGLOGLN

  • 1 – 999 999 999
  • OFF

MQIA_MEDIA_IMAGE_LOG_LENGTH (270)

  • 1 – 999 999 999
  • MQMEDIMGLOGLN_OFF (0)
Image Recover Object

IMGRCOVO

  • NO
  • YES

MQIA_MEDIA_IMAGE_RECOVER_OBJ (271)

  • MQIMGRCOV_NO (0)
  • MQIMGRCOV_YES (1)
Image Recover Queue

IMGRCOVQ

  • NO
  • YES

MQIA_MEDIA_IMAGE_RECOVER_Q (272)

  • MQIMGRCOV_NO (0)
  • MQIMGRCOV_YES (1)

Queue Manager Status

New Attribute MQSC name
See DISPLAY QMSTATUS
Look for KC 902 indicator
PCF constant and values
See Inquire Queue Manager Status
Look for KC 902 indicator
Archive Log Extent Name

ARCHLOG

MQCACF_ARCHIVE_LOG_EXTENT_NAME (3208)

  • String of length MQ_LOG_EXTENT_NAME_LENGTH (24)
Archive Log Size

ARCHSZ

MQIACF_ARCHIVE_LOG_SIZE (1416)

Media Log Size

MEDIASZ

MQIACF_MEDIA_LOG_SIZE (1417)

Restart Log Size

RECSZ

MQIACF_RESTART_LOG_SIZE (1418)

Reusable Log Size

REUSESZ

MQIACF_REUSABLE_LOG_SIZE (1419)

Archive Log In Use

LOGINUSE

MQIACF_LOG_IN_USE (1420)

Archive Log Utilization

LOGUTIL

MQIACF_LOG_UTILIZATION (1421)

Reset QMgr command

Updated attribute MQSC name
See RESET QMGR
Look for KC 902 indicator
PCF constant and values
See Reset Queue Manager
Look for KC 902 indicator
Action

TYPE

  • REDUCELOG
  • ARCHLOG

MQIACF_ACTION (1086)

  • MQACT_REDUCE_LOG (10)
  • MQACT_ARCHIVE_LOG (11)
Archived Log

ARCHIVED

MQCACF_ARCHIVE_LOG_EXTENT_NAME (3208)

  • String of length MQ_LOG_EXTENT_NAME_LENGTH (24)
Log Reduction

REDUCE

  • AUTO
  • ONE
  • MAX

MQIACF_LOG_REDUCTION (1422)

  • MQLR_AUTO (-1)
  • MQLR_ONE (1)
  • MQLR_MAX (-2)

Queue Local and Queue Model

New Attribute MQSC name
See DEFINE queues
Look for KC 902 indicator
PCF constant and values
See Change, Copy, and Create Queue
Look for KC 902 indicator
Image Recover Queue

IMGRCOVQ

  • NO
  • YES
  • QMGR

MQIA_MEDIA_IMAGE_RECOVER_Q (272)

  • MQIMGRCOV_NO (0)
  • MQIMGRCOV_YES (1)
  • MQIMGRCOV_AS_Q_MGR (2)

You can get the equivalent information for earlier Command Levels from these posts.