QLOAD version 9.0.3 – Reading messages from the log

MQGem Software is pleased to announce that a new version of QLOAD, our queue load/unload tool for IBM MQ, is now available.

This release was created for one specific customer requested feature, retrieving messages from your queue manager log.

IBM MQ comes with a utility, dmpmqlog, which can format your queue manager transactional log and dump out the contents of messages in a hex format (example below snipped for brevity).

LOG RECORD - LSN            
**********

HLG Header: lrecsize 720, version 1, rmid 0, eyecatcher HLRH

LogRecdType . . : AQM Put Message (257)
Eyecatcher  . . : ALRH                 Version . . . . : 1
LogRecdLen  . . : 700                  LogRecdOwnr . . : 256    (AQM)
XTranid . . . . : TranType: NULL
QueueName . . . : HELLO.WORLD.Q                               

Data  . . . . . : 
00000:  41 51 52 48 04 00 00 00 FF FF FF FF FF FF FF FF    AQRH....ÿÿÿÿÿÿÿÿ
:
00160:  4D 44 20 20 01 00 00 00 00 00 00 00 08 00 00 00    MD  ............
00176:  00 00 00 00 22 02 00 00 52 03 00 00 4D 51 53 54    ...."...R...MQST
00192:  52 20 20 20 00 00 00 00 01 00 00 00 20 20 20 20    R   ........    
:
00368:  20 20 20 20 20 20 20 20 0B 00 00 00 43 3A 5C 6D            ....C:\m
00384:  71 6D 38 30 30 34 5C 62 69 6E 36 34 5C 61 6D 71    qm8004\bin64\amq
00400:  73 70 75 74 2E 65 78 65 32 30 31 38 30 34 32 33    sput.exe20180423
00416:  30 34 33 34 33 34 35 36 20 20 20 20 00 00 00 00    04343456    ....
00432:  00 00 00 00 00 00 00 00 00 00 00 00 FF FF 00 00    ............ÿÿ..
00448:  48 65 6C 6C 6F 20 57 6F 72 6C 64 21                Hello World!

Log To QueueThis format is a little tricky to extract the data from to reinstate your message. As you can see above, you get the MQMD and the message data all in one block. It gets even more tricky when your message is broken up into several log records!

This version of QLOAD hopes to make the process of recreating message from your transactional log files much simpler to do. It can only work when the messages you want are in the log file though. There’s no magic!

Running QLOAD against the queue manager log

With the following invocation, QLOAD will run the dmpmqlog utility and parse through the output looking for the queue you requested, in this example HELLO.WORLD.Q, and write any messages it finds to the HELLO.WORLD.RCOV queue on queue manager MQG2.

qload -m MQG1 -j* -i HELLO.WORLD.Q -m MQG2 -o HELLO.WORLD.RCOV

Since you are reading the log files using dmpmqlog, the queue manager, MQG1, cannot be running when this command is issued and so the messages cannot be put onto a queue on that queue manager as part of the command. In this example I’ve chosen to put them onto a side queue on another queue manager but equally I could write them into a QLOAD file and then later put them onto the queue manager once it has started up again, as follows:-

qload -m MQG1 -j* -i HELLO.WORLD.Q -f HELLO_WORLD.qld
strmqm MQG1
qload -m MQG1 -f HELLO_WORLD.qld -o HELLO.WORLD.RCOV

Running QLOAD against output from dmpmqlog

If you have already run dmpmqlog, perhaps to determine whether the messages you wish to retrieve can be found in the log, then you may have already spent a few minutes waiting for the utility to complete. You can use the output files you generated from the utility as input into QLOAD.

dmpmqlog -m MQG1 -b > dmpmqlog.txt
qload -j dmpmqlog.txt -i HELLO.WORLD.Q -m MQG1 -o HELLO.WORLD.RCOV

Running QLOAD against copied off queue manager logs

A third alternative is to run QLOAD against queue manager log files that have been copied off. In this case you provide QLOAD with a directory containing these files.

qload -m MQG1 -j c:\mqmarch\log\MQG1 -i HELLO.WORLD.Q -m MQG1 -o HELLO.WORLD.RCOV

This command drives dmpmqlog with the -f parameter, and so the criteria for this directory must follow what is described in Knowledge Center for running dmpmqlog in that way. In addition, you must use the -m parameter to specific the queue manager name that these log files came from so that queue names can be found, and that queue manager cannot be running at the time.

The specified directory must contain the log header file (amqhlctl.lfh) and a subdirectory called active. The active subdirectory must contain the log files. By default, log files are assumed to be in the directories specified in the IBM MQ configuration information. If you use this option, queue names associated with queue identifiers are shown in the dump only if you use the -m option to name a queue manager name that has the object catalog file in its directory path.

QLOAD filtering

Whichever of the above methods you choose to use, you can add many of the QLOAD filtering options to your command.

For example, you can find all the messages containing a particular search string.

qload -m MQG1 -j* -i HELLO.WORLD.Q -s "World" -m MQG2 -o HELLO.WORLD.RCOV

Or perhaps you need to pluck out one specific message using its message id (or correlation id).

qload -m MQG1 -j* -i HELLO.WORLD.Q -gxm414D51204D51473120202020202020202604E15A21D49501 -m MQG2 -o HELLO.WORLD.RCOV

QLOAD also has time based filtering, so if you’re looking for a message from hours, days or weeks ago, you can go directly to that time period. This example is looking for messages that were put onto the queue between 18 and 19 hours ago.

qload -m MQG1 -j* -i HELLO.WORLD.Q -T0:18:00,0:19:00 -m MQG2 -o HELLO.WORLD.RCOV

Note: Filtering by Selection String (the -H parameter) is not available when your input source is a file (either a QLOAD file or an MQ log file) since it uses the Selector feature of MQOPEN.


The new version can be downloaded from the QLOAD Download Page. Any current licensed users of QLOAD can run the new version on their existing licence. If you don’t have a licence and would like to try out QLOAD then send an email to support@mqgem.com and a 1-month trial licence will be sent to you.

Advertisements

QLOAD version 9.0.2 – Required Rate

MQGem Software is pleased to announce that a new version of QLOAD, our queue load/unload tool for IBM MQ, is now available.

This release was created for one specific customer requested feature which we call Required Rate Processing.

Required Rate

QLOAD Source to TargetThis feature allows you to test the speed that your system, or specific applications in your system, can process messages. You may have a target rate you need to be able to achieve, and you can use QLOAD to send messages through the system at that rate to see whether the system (the channels and the applications) can keep up. You may for example have TLS channels; or message compression; or AMS message level encryption applied to the messages which are of course, changing the performance profile of the system. With QLOAD you can test the rate using exactly the same shape and size of messages that you will use in the production environment.

Starting with a queue that contains a representative selection of example messages, you can then use QLOAD to copy those messages and inject them into the system at the rate you require. In the example that follows, the required rate is 500 messages/second and the test is to run for 10 minutes.

qload -m MQG1 -i EXAMPLE.MSGS -o TARGET.QUEUE -R500:600

QLOAD will browse the EXAMPLE.MSGS queue and send copies of those messages to the TARGET.QUEUE. When it reaches the end of the EXAMPLE.MSGS then it will go back to the beginning of the queue and send them again, as often as required until the test ends. You can just have a single message on the queue and send it repeatedly, but to offset the cost of resetting the browse, it’s better to have 10 or more messages on the queue. In reality all your messages will not be the same, so a good selection of example messages is certainly worth collecting.

If you want the test to run until you tell it to stop, you can indicate that you want it to run indefinitely with an asterisk for the duration.

qload -m MQG1 -i EXAMPLE.MSGS -o TARGET.QUEUE -R500:*

There are two other ways you might use this feature.

Discover Achievable Rate

Instead of telling QLOAD to run at a certain rate, you might instead want to discover what your highest rate could be.

To run QLOAD in this way, you use a command like this:-

qload -m MQG1 -i EXAMPLE.MSGS -o TARGET.QUEUE -R*:*

And you will see output like the following, as QLOAD ramps up the rate.

Monitoring depth of queue 'TARGET.QUEUE'
Message Rate 802 msgs/sec
Message Rate 5200 msgs/sec
Message Rate 5548 msgs/sec
Message Rate 5516 msgs/sec
Message Rate 5409 msgs/sec
Message Rate 5566 msgs/sec
Message Rate 5503 msgs/sec

It does this by monitoring the depth of the target queue, which might be a transmission queue being read by a channel, or it might be an application queue being read either by the read application or by another instance of QLOAD acting as a sink.

qload -I TARGET.QUEUE -f null -w600

Trickle Feed the contents of a queue

So far you’ve seen QLOAD copy example messages from a source queue and inject them into the system over and over until the duration of the test is complete. Alternatively though, you could use this feature to move messages from a side queue (the contents of a batch run, or messages from a backout queue or a Dead-letter queue) back into the main system. If you have a large depth of these messages you may not want to load them back into the application queue all at once, since you know deep queues hurt performance. So instead you can trickle feed them into the system using QLOAD.

qload -m MQG1 -I SIDE.QUEUE -o TARGET.QUEUE -R50

The new version can be downloaded from the QLOAD Download Page. Any current licensed users of QLOAD can run the new version on their existing licence. If you don’t have a licence and would like to try out QLOAD then send an email to support@mqgem.com and a 1-month trial licence will be sent to you.