MQGem tools on Linux on Little Endian Power

MQGem is pleased to announce the availability of three of its products on the Linux on Little Endian Power platform.

We have had these products available on Linux on Power for a number of years, and now have added the Little Endian platform to the list.

Thanks to those customers who requested it.

You cannot run a Big Endian application on a Little Endian platform and vice versa. Ensure you download the appropriate version of the tool for the platform you are using. For example, if you try to run the Big Endian version on a Little Endian platform, you will see an error something like this:-

-bash: ./mqscx: cannot execute binary file

If you are not sure which platform you are using, try one of the following commands:-

lscpu

which will show you output similar to:

Architecture:          ppc64le
Byte Order:            Little Endian
CPU(s):                ...

Or if you have an older version of Linux, which doesn’t provide that command, try:-

uname -a

which might give you hint.

Linux gemlnx4.mqgem.com 3.10.0-327.36.3.el7.ppc64le #1 SMP Thu Oct 20 04:58:48 EDT 2016 ppc64le ppc64le ppc64le GNU/Linux

Extra Reading Material


The new versions can be downloaded from the website from the links given above. Any current licensed users of these products can run them on the new platform using their existing licence. If you don’t have a licence and would like to try out any of our products then send an email to support@mqgem.com and a 1-month trial licence will be sent to you.

Advertisements

QLOAD version 9.0.3 – Reading messages from the log

MQGem Software is pleased to announce that a new version of QLOAD, our queue load/unload tool for IBM MQ, is now available.

This release was created for one specific customer requested feature, retrieving messages from your queue manager log.

IBM MQ comes with a utility, dmpmqlog, which can format your queue manager transactional log and dump out the contents of messages in a hex format (example below snipped for brevity).

LOG RECORD - LSN            
**********

HLG Header: lrecsize 720, version 1, rmid 0, eyecatcher HLRH

LogRecdType . . : AQM Put Message (257)
Eyecatcher  . . : ALRH                 Version . . . . : 1
LogRecdLen  . . : 700                  LogRecdOwnr . . : 256    (AQM)
XTranid . . . . : TranType: NULL
QueueName . . . : HELLO.WORLD.Q                               

Data  . . . . . : 
00000:  41 51 52 48 04 00 00 00 FF FF FF FF FF FF FF FF    AQRH....ÿÿÿÿÿÿÿÿ
:
00160:  4D 44 20 20 01 00 00 00 00 00 00 00 08 00 00 00    MD  ............
00176:  00 00 00 00 22 02 00 00 52 03 00 00 4D 51 53 54    ...."...R...MQST
00192:  52 20 20 20 00 00 00 00 01 00 00 00 20 20 20 20    R   ........    
:
00368:  20 20 20 20 20 20 20 20 0B 00 00 00 43 3A 5C 6D            ....C:\m
00384:  71 6D 38 30 30 34 5C 62 69 6E 36 34 5C 61 6D 71    qm8004\bin64\amq
00400:  73 70 75 74 2E 65 78 65 32 30 31 38 30 34 32 33    sput.exe20180423
00416:  30 34 33 34 33 34 35 36 20 20 20 20 00 00 00 00    04343456    ....
00432:  00 00 00 00 00 00 00 00 00 00 00 00 FF FF 00 00    ............ÿÿ..
00448:  48 65 6C 6C 6F 20 57 6F 72 6C 64 21                Hello World!

Log To QueueThis format is a little tricky to extract the data from to reinstate your message. As you can see above, you get the MQMD and the message data all in one block. It gets even more tricky when your message is broken up into several log records!

This version of QLOAD hopes to make the process of recreating message from your transactional log files much simpler to do. It can only work when the messages you want are in the log file though. There’s no magic!

Running QLOAD against the queue manager log

With the following invocation, QLOAD will run the dmpmqlog utility and parse through the output looking for the queue you requested, in this example HELLO.WORLD.Q, and write any messages it finds to the HELLO.WORLD.RCOV queue on queue manager MQG2.

qload -m MQG1 -j* -i HELLO.WORLD.Q -m MQG2 -o HELLO.WORLD.RCOV

Since you are reading the log files using dmpmqlog, the queue manager, MQG1, cannot be running when this command is issued and so the messages cannot be put onto a queue on that queue manager as part of the command. In this example I’ve chosen to put them onto a side queue on another queue manager but equally I could write them into a QLOAD file and then later put them onto the queue manager once it has started up again, as follows:-

qload -m MQG1 -j* -i HELLO.WORLD.Q -f HELLO_WORLD.qld
strmqm MQG1
qload -m MQG1 -f HELLO_WORLD.qld -o HELLO.WORLD.RCOV

Running QLOAD against output from dmpmqlog

If you have already run dmpmqlog, perhaps to determine whether the messages you wish to retrieve can be found in the log, then you may have already spent a few minutes waiting for the utility to complete. You can use the output files you generated from the utility as input into QLOAD.

dmpmqlog -m MQG1 -b > dmpmqlog.txt
qload -j dmpmqlog.txt -i HELLO.WORLD.Q -m MQG1 -o HELLO.WORLD.RCOV

Running QLOAD against copied off queue manager logs

A third alternative is to run QLOAD against queue manager log files that have been copied off. In this case you provide QLOAD with a directory containing these files.

qload -m MQG1 -j c:\mqmarch\log\MQG1 -i HELLO.WORLD.Q -m MQG1 -o HELLO.WORLD.RCOV

This command drives dmpmqlog with the -f parameter, and so the criteria for this directory must follow what is described in Knowledge Center for running dmpmqlog in that way. In addition, you must use the -m parameter to specific the queue manager name that these log files came from so that queue names can be found, and that queue manager cannot be running at the time.

The specified directory must contain the log header file (amqhlctl.lfh) and a subdirectory called active. The active subdirectory must contain the log files. By default, log files are assumed to be in the directories specified in the IBM MQ configuration information. If you use this option, queue names associated with queue identifiers are shown in the dump only if you use the -m option to name a queue manager name that has the object catalog file in its directory path.

QLOAD filtering

Whichever of the above methods you choose to use, you can add many of the QLOAD filtering options to your command.

For example, you can find all the messages containing a particular search string.

qload -m MQG1 -j* -i HELLO.WORLD.Q -s "World" -m MQG2 -o HELLO.WORLD.RCOV

Or perhaps you need to pluck out one specific message using its message id (or correlation id).

qload -m MQG1 -j* -i HELLO.WORLD.Q -gxm414D51204D51473120202020202020202604E15A21D49501 -m MQG2 -o HELLO.WORLD.RCOV

QLOAD also has time based filtering, so if you’re looking for a message from hours, days or weeks ago, you can go directly to that time period. This example is looking for messages that were put onto the queue between 18 and 19 hours ago.

qload -m MQG1 -j* -i HELLO.WORLD.Q -T0:18:00,0:19:00 -m MQG2 -o HELLO.WORLD.RCOV

Note: Filtering by Selection String (the -H parameter) is not available when your input source is a file (either a QLOAD file or an MQ log file) since it uses the Selector feature of MQOPEN.


The new version can be downloaded from the QLOAD Download Page. Any current licensed users of QLOAD can run the new version on their existing licence. If you don’t have a licence and would like to try out QLOAD then send an email to support@mqgem.com and a 1-month trial licence will be sent to you.

Use MQGem tools with IBM MQ on IBM Cloud

You may have seen the recent announcement from IBM about the experimental IBM MQ service running on IBM Cloud.
IBM Cloud

You can learn more with these resources:-

When you read the description of this new service in the Bluemix catalog, you’ll see it says the following:-

Manage MQ your way

Manage your cloud-based queue managers with the tools you know and love – including MQ Explorer, the MQ Console, or via MQ Script Commands (MQSC).

This blog post is here to assure you that the tools you know and love from MQGem Software; MO71, MQSCX, MQEdit and QLOAD can all also be used with IBM MQ running on IBM Cloud.

Once you have created your MQ on IBM Cloud Service and Queue Manager, as shown in the above video, and your queue manager is up and running, you’ll have a view something like this.

IBM Cloud QM List

A list showing my Running queue manager

Click on the three vertical dots on the right of your queue manager to “Download Connection info”, or alternatively view the details of your queue manager and then there is a button there too which allows you to download the “Connection Information”. Either way, you’ll be presented with a pop-up which allows you to download a plain text file which contains the queue manager name, hostname, port number and a couple of channel names, one called an Application Channel and one called an Administration Channel.

IBM Cloud Download Connection info

IBM Cloud Download Connection info

In order to remotely connect to your IBM Cloud queue manager you will also need to be able to log in. As the MQ on IBM Cloud documentation describes here, you need the API key as your password to go with the user id ‘admin’. Follow the instructions on that page to obtain your API key.

Now you have all the pieces of information you need to set up any of the MQGem Software tools to administer your queue manager on IBM Cloud. As a reminder, these are the things you will need.

Item From where
Queue Manager Name You invented it when you created the queue manager. If you’ve forgotten it, it’s also in the text file you downloaded with the connection information.
Channel Name The “Administration channel name” can be found in the text file you downloaded with the connection information.
Connection Name This is built by concatenating the “Hostname” and “Listener port” details (with brackets round the port number) that can be found in the text file you downloaded with the connection information.
User ID This is ‘admin’
Password This is the API Key that you created by following the linked instructions.

On the pages that follow, we cover how to use the above information you have gathered in your table to configure each of our tools to connect to your IBM MQ in IBM Cloud Queue Manager. Go directly to the page for the tool you want to use, or page through each one in turn.

  1. MO71
  2. MQEdit
  3. MQSCX
  4. QLOAD

If you don’t have a licence and would like to try out any of our tools then send an email, noting which tool you’d like to try, to support@mqgem.com and a 1-month trial licence will be sent to you.

QLOAD version 9.0.2 – Required Rate

MQGem Software is pleased to announce that a new version of QLOAD, our queue load/unload tool for IBM MQ, is now available.

This release was created for one specific customer requested feature which we call Required Rate Processing.

Required Rate

QLOAD Source to TargetThis feature allows you to test the speed that your system, or specific applications in your system, can process messages. You may have a target rate you need to be able to achieve, and you can use QLOAD to send messages through the system at that rate to see whether the system (the channels and the applications) can keep up. You may for example have TLS channels; or message compression; or AMS message level encryption applied to the messages which are of course, changing the performance profile of the system. With QLOAD you can test the rate using exactly the same shape and size of messages that you will use in the production environment.

Starting with a queue that contains a representative selection of example messages, you can then use QLOAD to copy those messages and inject them into the system at the rate you require. In the example that follows, the required rate is 500 messages/second and the test is to run for 10 minutes.

qload -m MQG1 -i EXAMPLE.MSGS -o TARGET.QUEUE -R500:600

QLOAD will browse the EXAMPLE.MSGS queue and send copies of those messages to the TARGET.QUEUE. When it reaches the end of the EXAMPLE.MSGS then it will go back to the beginning of the queue and send them again, as often as required until the test ends. You can just have a single message on the queue and send it repeatedly, but to offset the cost of resetting the browse, it’s better to have 10 or more messages on the queue. In reality all your messages will not be the same, so a good selection of example messages is certainly worth collecting.

If you want the test to run until you tell it to stop, you can indicate that you want it to run indefinitely with an asterisk for the duration.

qload -m MQG1 -i EXAMPLE.MSGS -o TARGET.QUEUE -R500:*

There are two other ways you might use this feature.

Discover Achievable Rate

Instead of telling QLOAD to run at a certain rate, you might instead want to discover what your highest rate could be.

To run QLOAD in this way, you use a command like this:-

qload -m MQG1 -i EXAMPLE.MSGS -o TARGET.QUEUE -R*:*

And you will see output like the following, as QLOAD ramps up the rate.

Monitoring depth of queue 'TARGET.QUEUE'
Message Rate 802 msgs/sec
Message Rate 5200 msgs/sec
Message Rate 5548 msgs/sec
Message Rate 5516 msgs/sec
Message Rate 5409 msgs/sec
Message Rate 5566 msgs/sec
Message Rate 5503 msgs/sec

It does this by monitoring the depth of the target queue, which might be a transmission queue being read by a channel, or it might be an application queue being read either by the read application or by another instance of QLOAD acting as a sink.

qload -I TARGET.QUEUE -f null -w600

Trickle Feed the contents of a queue

So far you’ve seen QLOAD copy example messages from a source queue and inject them into the system over and over until the duration of the test is complete. Alternatively though, you could use this feature to move messages from a side queue (the contents of a batch run, or messages from a backout queue or a Dead-letter queue) back into the main system. If you have a large depth of these messages you may not want to load them back into the application queue all at once, since you know deep queues hurt performance. So instead you can trickle feed them into the system using QLOAD.

qload -m MQG1 -I SIDE.QUEUE -o TARGET.QUEUE -R50

The new version can be downloaded from the QLOAD Download Page. Any current licensed users of QLOAD can run the new version on their existing licence. If you don’t have a licence and would like to try out QLOAD then send an email to support@mqgem.com and a 1-month trial licence will be sent to you.

Migrating a Queue Manager?

If you’re planning to migrate a queue manager, from one machine to another, or perhaps you’re consolidating some queue managers onto an MQ Appliance, you are probably aware of, and have even become familiar with, the steps required to export your object definitions and recreate them on the new queue manager. There are a number of tools available that can do this for you – including those from MQGem.

What about your messages though?

While it’s generally a good idea to reduce the load, and drain off as many messages as you can prior to migrating a queue manager. For example if it’s in a cluster, suspend it from the cluster before the move, to reduce the number of messages heading its way. Let all the applications drain the queues of messages and so on. However, it’s not always possible to drain every last message from all your queues.

QLOAD Offload all queuesIn the latest version of QLOAD, V9.0.1, there’s a new feature that will help out when migrating a queue manager. It allows you to unload the messages from all your queues, with one command. And then you can load the messages onto the queues on another queue manager, with one command (or piecemeal if you prefer). You would combine this with your favoured object definition export tool, and before loading the messages, you’d first recreate the queues on the new queue manager with the exported commands.

Here’s an example of QLOAD unloading all the queues on my queue manager.

qload -m MQG1 -i* -f*

The command will produce the following output to show what has been unloaded. Using the -i (lower case) flag means that the messages are only browsed on the queues and are not destructively removed.

APP1.INPUT                                           2  Done.
APP1.RESULT                                         10  Done.
APP2.INPUT                                           1  Done.
APP2.RESULT                                          4  Done.
Q1                                                  28  Done.
Q2                                                  42  Done.
SYSTEM.ADMIN.QMGR.EVENT                              7  Done.
SYSTEM.AUTH.DATA.QUEUE                             126  Done.
SYSTEM.CHANNEL.SYNCQ                                 3  Done.
SYSTEM.CHLAUTH.DATA.QUEUE                            5  Done.
SYSTEM.CLUSTER.REPOSITORY.QUEUE                      3  Done.
SYSTEM.DURABLE.SUBSCRIBER.QUEUE                      1  Done.
SYSTEM.HIERARCHY.STATE                               2  Done.
SYSTEM.INTER.QMGR.FANREQ                             1  Done.
SYSTEM.RETAINED.PUB.QUEUE                            2  Done.
WORK.REQUEST                                         7  Done.

Total : 7 Queues, 94 Messages
 plus : 9 System Queues, 150 Messages

Listing the directory where I ran the qload command, I can now see I have a file for each queue, with an extension .qld. If you prefer to have a different extension then you can alter the command accordingly. For example, use -f*.txt.

I can copy these files to another machine, or simply use a client connection to the other machine accordingly. Then I can run the following QLOAD command to load the messages onto the new queue manager.

qload -m MQG1 -o* -f*

QLOAD will make some checks when you run a load for multiple queues in this way. It will check that all the queues exist that it has files for in the directory matching the file pattern you specified (which assumes an extension of .qld if you just use ‘*’), and it will check that all those queues are empty. If it finds any problems it will report as follows:-

APP1.RESULT                            RC(2085) Unknown object name.
APP2.RESULT                            Not empty.

There are potential problems with these queues.
Are you sure you want to continue?

This lets you know to go and define the missing queues if you need them – there are messages to go onto them so the assumption is that you do need them. It also warns you of non-empty queues. Now if you’ve already started using this queue manager, you might be expecting this situation, but otherwise, you should rectify it, and then re-run QLOAD.

APP1.INPUT                                       2     Done.
APP1.RESULT                                      10    Done.
APP2.INPUT                                       1     Done.
APP2.RESULT                                      4     Done.
Q1                                               28    Done.
Q2                                               42    Done.
WORK.REQUEST                                     7     Done.

Total : 7 Queues, 94 Messages

You’ll notice that this output does not mention the SYSTEM queues that were offloaded. A generic upload will not upload most SYSTEM queues.
SYSTEM.CLUSTER.TRANSMIT.QUEUE and SYSTEM.DEAD.LETTER.QUEUE are the exceptions. The messages were unloaded to files though, so if you really need to load them you can do so the traditional way, by specifying the full queue name on the command and not using the generic upload.


The new version can be downloaded from the QLOAD Download Page. Any current licensed users of QLOAD can run the new version on their existing licence. If you don’t have a licence and would like to try out QLOAD then send an email to support@mqgem.com and a 1-month trial licence will be sent to you.

QLOAD version 9.0.1 is released

MQGem Software is pleased to announce that a new version of QLOAD, our queue load/unload tool for IBM MQ, is now available.

The main features of the release are as follows:-

  • Unload and Load all queue with one command
    Great for taking a backups of messages or for migratory purposes. Save time and effort by doing all queues at once. Read more about this in Migrating a Queue Manager?.
  • Filter messages based on the reason code in the Dead-letter Header
    Very useful when dealing with messages on your Dead-letter Queue. Save off only those specific messages you are interested in.
  • Copy/Move messages based on the target Q/QMgr in a MQDLH or MQXQH
    Copy or move messages currently on the Dead-letter queue or a transmission queue, based on the target queue and/or queue manager in the MQDLH (Dead-letter header) or MQXQH (Transmission header).

The new version can be downloaded from the QLOAD Download Page. Any current licensed users of QLOAD can run the new version on their existing licence. If you don’t have a licence and would like to try out QLOAD then send an email to support@mqgem.com and a 1-month trial licence will be sent to you.

QLOAD version 8.0.2 is released

MQGem Software is pleased to announce that a new version of QLOAD, our queue load/unload tool for IBM MQ, is now available.

The main features of the release are as follows:-

  • The ability to target multiple destination queues
    It is now possible to target multiple MQ queues and these can now be qualified by Queue Manager name. This means that QLOAD can now be used as a queue replicator.
  • The ability to provide the destination names in a single file
    Going hand in hand with the above feature, if you have a large number of queues you wish to distribute messages to it can be more convenient to put the names of the queues in a file and just refer to that file. This is now possible in QLOAD with the Destination file.
  • Various new verbose options such as
    • A progress indicator of how many messages have been processed
    • Which MQI verbs are being used and their reason codes.
    • The list of targeted destinations.

The new version can be downloaded from the QLOAD Download Page. Any current licensed users of QLOAD can run the new version on their existing licence. If you don’t have a licence and would like to try out QLOAD then send an email to support@mqgem.com and a 1-month trial licence will be sent to you.